# AU News Monitor — ABC, news.com.au, AFR, SMH (`he_work/au-news-monitor`) Actor

Multi-source AU news: ABC News, news.com.au (DataDome bypass via curl_cffi), AFR, and SMH (RSS). Features anti-bot evasion, paywall-respecting extraction, keyword filtering, cross-run deduplication, and 365-day trend analysis.

- **URL**: https://apify.com/he_work/au-news-monitor.md
- **Developed by:** [HE Work](https://apify.com/he_work) (community)
- **Categories:** News, Automation, AI
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.28 / 1,000 articles

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AU News Monitor — ABC, news.com.au, AFR, SMH

> ✅ **Checked daily.** An automated test runs this Actor on the Apify cloud every day and checks that it still returns real data, so if the website changes and something breaks, it's caught the same day and fixed.

[![Python](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org)

**Multi-source Australian news aggregation from ABC News, news.com.au, Australian Financial Review (AFR), and Sydney Morning Herald (SMH).** Includes keyword filtering, cross-run deduplication, and trend analysis.

***

### Quick Overview

This actor monitors four major Australian news sources, extracting headlines, URLs, publication dates, and article snippets. It automatically deduplicates articles across runs (so you never see the same article twice) and optionally tracks keyword trends over time.

**What you get:**

- Headlines, URLs, publication dates, and snippets from all four sources
- Automatic category inference (politics, business, sport, technology, etc.)
- **Cross-run deduplication** — never see duplicate articles between runs
- **Trend analysis** — track keyword frequency and category distribution over up to 365 days
- Paywall-respecting — only publicly visible content is extracted

**Use cases:**

- Media monitoring and PR tracking
- Market intelligence and competitor research
- News aggregation services
- Research and academic data collection

***

### Data Schema

| Field | Type | Description |
|-------|------|-------------|
| `headline` | string | Article headline/title |
| `url` | string | Article URL |
| `source` | string | `abc`, `news.com.au`, `afr`, or `smh` |
| `published` | string | Publication date (ISO 8601) |
| `author` | string | Article author (if available) |
| `snippet` | string | Lead paragraph or description |
| `category` | string | Inferred category: politics, business, sport, technology, world, entertainment, health, property, opinion, general |
| `word_count` | integer | Word count of snippet |
| `scraped_at` | string | When article was scraped (ISO 8601) |

**Example output:**

```json
{
  "headline": "RBA holds interest rates steady for fifth consecutive month",
  "url": "https://www.abc.net.au/news/2026-05-13/rba-interest-rates-hold/...",
  "source": "abc",
  "published": "2026-05-13T14:30:00+10:00",
  "category": "business",
  "snippet": "The Reserve Bank has left the cash rate unchanged at 3.85 per cent..."
}
```

***

### Input Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `sources` | string\[] | `["abc", "news.com.au", "afr", "smh"]` | News sources to scrape |
| `keywords` | string\[] | `[]` | Filter articles by keywords (word-boundary matching) |
| `categories` | string\[] | `[]` | Filter by category: politics, business, sport, etc. |
| `maxPerSource` | number | `0` | Max articles per source (0 = unlimited) |
| `includeTrends` | boolean | `false` | Enable keyword trend analysis |
| `trendKeywords` | string\[] | `[]` | Keywords to track trends for |

***

### Output

The actor provides:

1. **Article records** — individual articles with all metadata
2. **Deduplication** — articles previously seen in prior runs are excluded
3. **Trend analysis** (optional) — keyword frequency and category distribution with up to 365-day history

Dataset views in Apify Console: Overview, By Category, Trends, and Full Detail. Export as JSON, CSV, XML, or Excel.

***

### What Makes This Actor Different

- **Only AU-specific multi-source news actor** on Apify Store
- **Cross-run deduplication** — schedule daily and never see duplicate articles
- **Trend analysis** — track what topics are rising or falling in Australian media
- **Paywall-respecting** — only publicly visible content is extracted
- **Extremely low cost** — lightweight HTTP-based scraping, no browser required for most sources

***

### Known Limitations

| Limitation | Details |
|-----------|---------|
| AFR headlines only | Australian Financial Review has a hard paywall — article bodies are not extracted |
| SMH via RSS | Sydney Morning Herald uses their RSS feed (~20 most recent articles) — limited archive |
| Keyword filtering is post-scrape | All articles are collected first, then filtered — may return 0 results for narrow keywords |
| Trend data accumulates over time | Best results after multiple scheduled runs |

***

### Example output

One item from a real run (long texts shortened):

```json
{
  "headline": "Rugby league great Paul Langmack dies aged 61",
  "url": "https://www.abc.net.au/news/2026-11-05/rugby-league-great-paul-langmack-dies-aged-61/107348920",
  "source": "abc",
  "published": "2026-11-05T00:00:00",
  "author": null,
  "snippet": null,
  "category": "sport",
  "word_count": null,
  "scraped_at": "2026-11-05T07:24:56+00:00"
}
```

### Alerts & workflows

Ready-made workflow: **News monitor → Telegram / Slack**. No Make or Zapier needed — set it in the input and schedule the Actor (e.g. daily).

1. Give the run a **Watch list name** (e.g. `my-watch`). Every run with that name remembers every article URL it has seen; each item gets `change_status` = `new`, `changed` or `unchanged` (changed items also carry `previous_value`).
2. Turn on **Only new / changed items** to skip — and not pay for — items you've already seen.
3. Add one or more destinations; after each run you get a digest of what's new or changed:
   - **Slack**: an incoming-webhook URL.
   - **Telegram**: bot token (from @BotFather) + chat ID.
   - **Discord**: a channel webhook URL.
   - **Email**: an address (sent with Apify's send-mail Actor).
   - **Google Sheets**: open your sheet → Extensions → Apps Script, paste the script below, Deploy → New deployment → Web app (execute as you, access: anyone), and paste the web app URL into the input. Each new / changed item is appended as a row.

```javascript
function doPost(e) {
  var rows = JSON.parse(e.postData.contents).rows || [];
  var sheet = SpreadsheetApp.getActiveSpreadsheet().getActiveSheet();
  if (!rows.length) return ContentService.createTextOutput("0");
  var head = sheet.getLastRow() ? sheet.getRange(1, 1, 1, sheet.getLastColumn()).getValues()[0] : [];
  if (!head.length) { head = Object.keys(rows[0]); sheet.appendRow(head); }
  rows.forEach(function (r) { sheet.appendRow(head.map(function (h) { return r[h] === undefined ? "" : r[h]; })); });
  return ContentService.createTextOutput(String(rows.length));
}
```

Secrets (webhook URLs, bot tokens) are stored encrypted by Apify.

### How it compares

| | This Actor | Typical alternative |
|---|---|---|
| Fields per result | 11 (see *Example output*) | Varies by actor |
| Freshness | Read live from the source when the run starts | Varies |
| Reliability | Tested daily by an automated run on the Apify cloud (see the top of this page) | — |
| Price | $1.50 per 1,000 articles | $2.00–$8.00 per 1,000 results (comparable Actors on the Apify Store, 6 Nov 2026) |

### Pricing

Pay per result, no monthly fee:

- **$1.50 per 1,000 articles**: One news article delivered to your dataset.
- $0.0005 per run (start).

Bronze plans pay 5% less, Silver 10% less, Gold and above 15% less.

**Examples:** 1,000 articles = $1.50; 10,000 = $15.00.

***

### Who Is This For?

- **PR and media monitoring** professionals tracking brand mentions
- **Market intelligence** teams monitoring industry news
- **Researchers** collecting Australian media data
- **Journalists** tracking story development across outlets

***

### FAQ

**Does this bypass paywalls?** No — we extract only publicly visible content. AFR articles return headlines and URLs only, respecting the paywall. SMH uses their freely available RSS feed.

**How does deduplication work?** Each article URL is hashed and stored. On subsequent runs, previously seen articles are automatically excluded — you only get new content.

**What's trend analysis?** Enable `includeTrends` to track how often specific keywords appear over time. Trends accumulate across runs (up to 365 days of history).

**Do I need a proxy?** Most sources work without a proxy. For news.com.au, residential proxies improve reliability.

***

*Built on the Apify platform. Part of the AU Data Suite — Australian web data for developers and businesses.*

# Actor input Schema

## `sources` (type: `string`):

Comma-separated: abc,news,afr,smh

## `keywords` (type: `string`):

Search keywords

## `maxArticles` (type: `number`):

Max articles per source

## `watchListName` (type: `string`):

Any name, e.g. 'qld-tenders'. Runs with the same name compare against each other: items get change_status new / changed / unchanged.

## `onlyNewItems` (type: `boolean`):

With a watch list name: skip (and don't charge for) items that were already seen and haven't changed.

## `slackWebhookUrl` (type: `string`):

Slack incoming-webhook URL. Gets a digest of new / changed items after each run.

## `telegramBotToken` (type: `string`):

From @BotFather. Use together with the chat ID.

## `telegramChatId` (type: `string`):

Chat, group or channel ID that receives the digest (e.g. -1001234567890).

## `discordWebhookUrl` (type: `string`):

Discord channel webhook URL. Gets the digest.

## `googleSheetsWebAppUrl` (type: `string`):

Deploy the 10-line Apps Script from the README in your sheet and paste its web app URL: every new / changed item is appended as a row.

## `notifyEmail` (type: `string`):

Email address for the digest (sent with Apify's send-mail Actor).

## Actor input object example

```json
{
  "sources": "abc,news,afr,smh",
  "keywords": "",
  "maxArticles": 5,
  "watchListName": "my-watch-list",
  "onlyNewItems": false,
  "notifyEmail": "you@example.com"
}
```

# Actor output Schema

## `articleResults` (type: `string`):

Dataset containing all articles with headlines, sources, categories, and trend analysis data

## `trendHistory` (type: `string`):

Key-value store containing URL hash deduplication data and historical trend snapshots (up to 365 days)

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("he_work/au-news-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("he_work/au-news-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call he_work/au-news-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,he_work/au-news-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/VxTmMgbpeQcJVjzYF/builds/FUOb7HqeA7PEacvaV/openapi.json
