Google News Scraper — real publisher URLs, monitoring & full text

Scrape Google News by keyword or topic feed. Each item includes the headline, publisher, publish time, snippet, related coverage, the Google News link, and — when enabled — the real publisher URL and optional full article text.

Plain HTTP (no browser). Works with Google News RSS, decodes article IDs through Google’s batchexecute endpoint, and can page past the usual ~100-result cap by slicing the date window day by day.

Why this scraper

• Real publisher URLs — not only news.google.com redirect links
• Past the 100-result limit — day slicing with after: / before: when you ask for more than 100 items or a date range
• Only-new monitoring — remember seen article IDs for scheduled runs; pay only for new items
• Optional full text — JSON-LD articleBody or <article> paragraphs from the publisher page
• Pay per result — charge only for articles (and full texts) actually saved

Pricing

• $1.50 per 1,000 articles ($0.0015 each)
• +$1.50 per 1,000 full texts when full-text extraction succeeds

No monthly rental. Set a max total charge on the run; the actor stops cleanly and keeps everything already saved.

Input

Field	What it does	Default
queries	Keywords; Google operators allowed (when:1d, site:cnbc.com, …)	—
topics	TOP_STORIES, WORLD, NATION, BUSINESS, TECHNOLOGY, ENTERTAINMENT, SPORTS, SCIENCE, HEALTH	—
language / country	Google hl / gl (ceid derived automatically)	en-US / US
timeRange	any, 1h, 1d, 7d, 30d, 1y → when: on queries	7d
dateFrom / dateTo	YYYY-MM-DD; override timeRange	—
maxArticlesPerQuery	Cap per query/topic (1–5000). Over 100 triggers day slicing	100
decodeUrls	Resolve real publisher URLs	true
fetchFullText	Download publisher page body	false
onlyNew + monitorKey	Skip IDs stored under this monitor name	false / default
proxyConfiguration	Optional proxy; retries rotate session	none
requestTimeoutSecs	Per-request timeout	30

Example — one-off search

{
  "queries": ["artificial intelligence", "site:reuters.com climate"],
  "timeRange": "7d",
  "maxArticlesPerQuery": 50,
  "decodeUrls": true,
  "fetchFullText": false
}

Example — scheduled monitoring (only new)

{
  "queries": ["tesla"],
  "timeRange": "1d",
  "maxArticlesPerQuery": 100,
  "onlyNew": true,
  "monitorKey": "tesla-daily",
  "decodeUrls": true
}

Run on a schedule. The first run saves matches; later runs skip IDs already stored in the google-news-monitor key-value store under seen-tesla-daily (up to 20 000 recent IDs).

Example — topic feed + full text

{
  "topics": ["TECHNOLOGY"],
  "country": "US",
  "language": "en-US",
  "maxArticlesPerQuery": 30,
  "fetchFullText": true
}

Output (one item per article)

{
  "position": 1,
  "title": "Tesla’s self-driving Robotaxi service expands into greater Houston",
  "publisher": "Houston Public Media",
  "publisherUrl": "https://www.houstonpublicmedia.org",
  "url": "https://www.houstonpublicmedia.org/articles/news/transportation/2026/10/26/565273/teslas-robotaxi-houston-service-expands/",
  "googleNewsUrl": "https://news.google.com/rss/articles/CBMi…",
  "urlDecoded": true,
  "publishedAt": "2026-10-26T18:04:00.000Z",
  "publishedTimestamp": 1761501840000,
  "snippet": null,
  "relatedCoverage": [],
  "source": { "type": "query", "value": "tesla" },
  "language": "en-US",
  "country": "US",
  "fullText": "Optional article body…",
  "author": "Jane Doe",
  "image": "https://…",
  "scrapedAt": "2026-10-27T16:00:00.000Z"
}

RUN_SUMMARY in the default key-value store lists per-source found/saved/decoded counts and errors.

FAQ

Why is url null? Decoding failed for that item (layout change, block, or timeout). googleNewsUrl is still present and urlDecoded is false. The run continues.

Why fewer than maxArticlesPerQuery? Google may not have that many matching stories in the window, or day slices returned duplicates (deduped by article ID).

Does onlyNew cost anything for duplicates? No. Seen IDs are skipped before push/charge.

Full text empty? Many sites block bots or ship body text only via JavaScript. You still get metadata; fullTextError explains the failure. Full-text events are charged only when text was obtained.

Limitations

• Google News RSS returns at most ~100 items per request; higher limits rely on day slicing (day is the smallest slice).
• Publisher full-text quality varies; paywalled or JS-heavy pages often yield no body.
• Decode and full-text need outbound HTTP; use a proxy if your environment is blocked.
• Not affiliated with Google. Public news listings only; no login or personal data.