Google News Scraper — New Articles Only + RSS/Atom

A Google News scraper and RSS/Atom monitor that returns only new articles since your last run. Point it at Google News keyword/topic queries and/or public feed URLs, give each monitor a stable monitorId, and every subsequent run delivers canonical, deduped article metadata you have not seen before — ready for an LLM, agent, webhook, or spreadsheet.

Why delta

Pull scrapers and raw feed readers re-send the same headlines every poll. You end up building your own seen-set, URL normalizer, and cross-source dedup. This actor keeps a persistent per-monitorId seen-set in a named key-value store and returns the delta only: title, canonical publisher URL, source domain, published date, and snippet. No publisher full text (copyright + scope). By default, the first run records a baseline and delivers no article rows. It has no article fee, but start and successful-poll fees still apply. Set deliverOnFirstRun: true to receive existing articles on that run and pay the article fee too.

Quickstart (Console)

1. Open the actor → Input.
2. Keep the prefill query artificial intelligence or add your own queries / feeds.
3. Set monitorId (e.g. ai-watch) so later runs share the same seen-set.
4. Start — first run writes a baseline row. Run again with the same monitorId to receive article rows.

Input reference

Field	Type	Default	Notes
queries	string list	prefill ["artificial intelligence"]	Google News search queries; supports site:, quotes, OR, when:7d
feeds	string list	[]	RSS 2.0 / Atom / JSON Feed URLs (http(s), public hosts only)
monitorId	string	"default"	Identity of the persistent seen-set
maxArticlesPerQuery	integer	50 (max 100)	Newest-first cap per Google News query
maxTotalNewArticles	integer	500	Cap on delivered article rows per run
language	string	"en"	Google News hl
country	string	"US"	Google News gl; ceid = country:language
deliverOnFirstRun	boolean	false	If true, first run delivers + charges like a normal delta run

Output contract

Four row types in the default dataset:

article (charged as new-article)

{
    "type": "article",
    "monitorId": "ai-watch",
    "sourceType": "query",
    "query": "artificial intelligence",
    "title": "Example headline",
    "url": "https://publisher.example/story",
    "urlIsGoogleRedirect": false,
    "sourceDomain": "publisher.example",
    "publishedAt": "2026-08-01T12:00:00.000Z",
    "snippet": "Plain-text snippet up to 500 chars",
    "firstSeenAt": "2026-08-01T13:00:00.000Z"
}

baseline (no row fee; start and successful-poll fees apply)

{
    "type": "baseline",
    "monitorId": "ai-watch",
    "sources": [{ "source": "artificial intelligence", "sourceType": "query", "status": "ok", "itemsFetched": 50, "newItems": 50 }],
    "message": "Baseline recorded — subsequent runs deliver only new articles."
}

summary (uncharged, always last)

{
    "type": "summary",
    "monitorId": "ai-watch",
    "baseline": false,
    "runCount": 2,
    "sources": [{ "source": "artificial intelligence", "sourceType": "query", "status": "ok", "itemsFetched": 50, "newItems": 3 }],
    "totals": { "sourcesPolled": 1, "newArticles": 3, "duplicatesSkipped": 47, "decodeFailures": 0 },
    "chargeLimitReached": false
}

error (uncharged)

{
    "type": "error",
    "error": "no-sources",
    "message": "Provide at least one Google News query in `queries` or one feed URL in `feeds`.",
    "monitorId": "default"
}

Per-source failures (sourceType: "query"|"feed") also emit error rows; the run still succeeds. Other run-level error codes (also uncharged, run still succeeds): no-sources, concurrent-run (another run currently holds this monitorId's lease — nothing polled or charged), state-read-failed / state-write-failed (monitor state store temporarily unavailable — nothing polled or charged).

monitorId + baseline

• Same monitorId → shared seen-set across runs (and schedules).
• First run with a new monitorId (runCount === 0) and deliverOnFirstRun: false: polls all sources (charges query-poll), records every item into seen state, delivers zero article rows and zero new-article charges, pushes one baseline row.
• The baseline is completed per source. A source that failed, or was skipped for budget or time, was never read, so its existing catalogue was never recorded — it is baselined on a later run without new-article charges; that run's start and successful-poll fees still apply. Sources that did complete move to delta billing immediately, so one broken feed never silences the rest of the monitor. While any source is still unbaselined the baseline/summary rows carry baselineIncomplete: true.
• Later runs deliver only identities not in the seen-set (URL key, Google News id, or title hash within 14 days). Items with no title are identified by URL alone, so several untitled items from one feed are never collapsed into a single row.
• An item whose only identifier is an opaque <guid> or JSON Feed id (urn:uuid:…, tag:…, a bare number) carries no link, so it is skipped rather than delivered — and billed — with a URL that cannot be opened.
• Only one run at a time may hold a given monitorId. The lease is a server-side lock, so the second run exits successfully with a concurrent-run error row and charges nothing — two runs can never sell you the same article. A run that crashes releases its monitor within about 8 minutes; a run that is still alive keeps it. Schedule a monitor no more often than its runs take to finish.

Recipes

Schedule every 30 minutes

Apify Schedules → this actor, interval 30 minutes, payload:

{
    "queries": ["artificial intelligence", "site:reuters.com chips"],
    "feeds": ["https://techcrunch.com/feed/"],
    "monitorId": "ai-watch",
    "maxArticlesPerQuery": 50
}

API (run-sync-get-dataset-items)

curl -s "https://api.apify.com/v2/acts/kaz_kakyo~google-news-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "content-type: application/json" \
  -d '{"queries":["artificial intelligence"],"monitorId":"api-demo"}'

The first call records a baseline (start and successful-poll fees apply). Later calls return articles only if the source has items the monitor has not already seen, plus a summary. To receive articles immediately, add "deliverOnFirstRun": true to the first call.

MCP / agents

Works with the Apify MCP server — agents can call this actor as a tool with the same JSON input. Use a stable monitorId per watch topic so the tool returns deltas instead of full dumps.

Pricing

Pay-per-event prices in USD, verified October 5, 2026. Check the Store pricing tab for current rates before running.

Event	Meaning
apify-actor-start	$0.00005 per start event; one event per GB of allocated memory, minimum one. At the current 1 GB default: $0.00005 per validated run.
query-poll	$0.001 per source (query or feed) successfully polled — 200 and 304 both count ($1 / 1,000 polls).
new-article	$0.001 per delivered article row ($1 / 1,000 articles).

A quiet run at 1 GB that successfully polls one source and delivers no articles costs $0.00105: $0.00005 start + $0.001 poll. A baseline with one successful source has the same event total. A run with one successful source and 10 delivered articles costs $0.01105. These examples assume the full listed events are charged; they exclude taxes and any separate account charges. Failed fetches and SSRF-blocked feeds have no poll fee, but a validated run may still incur its start fee. Invalid empty input charges nothing.

Set the run's maximum total charge in Apify to cap spending. maxTotalNewArticles caps article delivery, not start or poll fees; fewer sources and a suitable schedule reduce polling costs.

Honest limits

• Google News returns up to ~100 items per query; we cap with maxArticlesPerQuery.
• Canonical-URL decode uses a Google internal batchexecute path that can drift; on failure we still deliver the news.google.com redirect link with urlIsGoogleRedirect: true.
• No article full text — metadata and snippets only.
• Fuzzy title dedup uses a 14-day window (generic headlines can collide).
• Seen-set retention per monitor: 60 days, capped at 30,000 URL + 30,000 Google News + 10,000 title identities (oldest evicted first). An article resurfacing after falling out of that window is treated as new again.
• Each feed has a 20-second total fetch budget (redirects included) and a 5 MB body cap.
• Feeds must be publicly reachable; private/internal hosts (localhost, RFC1918, link-local, cloud metadata) are blocked by the SSRF guard. URLs with embedded user:pass@ credentials are rejected — use a token query parameter if your feed needs auth.
• Authenticated feed URLs are never echoed back in full. Wherever a feed URL appears in output (article rows, error rows, summaries, logs) its query-parameter values are replaced with *** — parameter names, host and path are kept, so https://host/feed.xml?edition=uk&api_key=SECRET is reported as https://host/feed.xml?edition=***&api_key=***. Credential-looking path segments (/reset/SECRET/…) and any user:pass@ are masked the same way.

Local development

npm install
node src/unit-check.js
apify run --purge -i '{"queries":["artificial intelligence"],"monitorId":"local-test"}'