Google News Scraper

Pulls Google News across 47 country/language editions — top stories, keyword search, the eight topic sections, and local news by place — and turns Google's opaque redirect links into real publisher article URLs.

Surfaces	Top stories · keyword search · 8 topics · local news by place · raw topic IDs · publisher IDs
Editions	47 country/language editions, from US:en to JP:ja, ID:id, BR:pt-419, EG:ar
Returns	headline, publisher, published time, edition — plus the resolved publisher URL and domain
Method	HTTP only. No API key, no login, no browser
Anti-bot	None on this surface — every feed answers 200 cold over plain TLS

The bit that actually matters: real article URLs

Every link in a Google News feed is an opaque token like CBMiswFBVV95cUxQcDN2djctWGl2.... It is not the publisher's URL, and it cannot be decoded offline — base64-decoding it just yields a second opaque token. The pre-2024 trick that unpacked a plain URL is dead.

This actor resolves it properly, HTTP-only, and gives you:

"articleUrl":    "https://www.politico.com/news/2026/09/08/supreme-court-rejects-...",
"articleDomain": "www.politico.com",
"googleNewsUrl": "https://news.google.com/rss/articles/CBMinwFBVV95cUxOUlZobkox..."

It is not free. Google embeds the per-article signature at the very end of a ~600 KB page, ignores Range requests, and puts no signatures on any listing page — so resolution costs one ~600 KB fetch per article (gzipped on the wire, fetched concurrently). Set resolveArticleUrls: false for fast headline-only sweeps; you keep the publisher's name and homepage, just not the article link.

Example input

{
  "includeTopStories": true,
  "queries": ["artificial intelligence", "\"supply chain\" when:7d"],
  "topics": ["TECHNOLOGY", "BUSINESS"],
  "locations": ["Jakarta"],
  "editions": ["US:en", "GB:en", "ID:id"],
  "resolveArticleUrls": true,
  "maxItemsPerFeed": 50
}

Every selection is fetched once per edition, so the run above is 5 feed specs × 3 editions = 15 feeds.

Search operators that work

Verified against the live service — these genuinely filter:

Operator	Example	Note
exact phrase	"tesla model y"
OR	tesla OR rivian
exclude	tesla -musk
site:	tesla site:reuters.com	collapses to a single publisher
intitle:	intitle:tesla
when:	tesla when:1h, when:7d	the cheapest way to slice a big topic
after: / before:	tesla after:2026-09-01 before:2026-09-05

allinurl: is not supported — it returns zero results rather than an error.

Output

Every row carries the portfolio envelope — _input, _source, _scrapedAt, recordType. Three record types share the dataset:

• ARTICLE — one per item: headline (with Google's - Publisher suffix stripped, raw kept), publisher, published time, edition, and the resolved articleUrl when resolution ran.
• FEED_SUMMARY — one per feed × edition: how many items the feed held, how many were emitted, how many URLs resolved or failed, plus requestsUsed and bytesDownloaded so the cost of a run is visible in the data rather than guessed.
• ERROR — one per feed that failed, so every input maps to at least one row.

Known limits — read these, they are structural

• There is no pagination. At all. num, count, start, page, max-results and n are every one of them ignored — all return the identical body. A feed gives what it gives (~38 top stories, ~100 search, 50–70 topic, 20–90 local) and that is the ceiling for one request. Breadth comes from more feeds, not deeper paging: add editions, split a query with when:/ after:, or search narrower terms. maxItemsPerFeed can only reduce results.
• No article body. Google News is an index of other people's journalism; the text lives on the publisher's site, behind their own paywalls and bot rules. This actor gives you the link, not the article.
• No Danish edition. Google does not have one. Every Danish URL spelling returns the Norwegian feed with <language>no</language>, so DK:da is deliberately excluded rather than shipped as a lookalike.
• Two upstream failures look like success, and are caught: an unrecognised topic returns 200 with 1.68 MB of HTML, and an unrecognised place returns 200 with a one-item "This feed is not available." stub. Both become ERROR rows.
• A search with no matches is not an error — it is returnedCount: 0 with a summary row.

Licensing — please read before publishing anything built on this

Google's robots.txt puts ClaudeBot, anthropic-ai and Claude-Web under a bare Disallow: /, and the feed carries its own terms in a <copyright> element: made available "solely for the purpose of rendering Google News results within a personal feed reader for personal, non-commercial use. Any other use of the feed is expressly prohibited."

Unlike a publisher API there is no commercial tier to buy — Google retired the Google News API in 2011. Those terms are Google's position on this data, they are not enforced technically, and anyone running this actor should decide for themselves whether their use fits. It is stated here plainly rather than left for you to find later.