Google News Scraper — keyword & company news alerts with full text

Google News Scraper turns any Google News search into clean, structured JSON: headline, publisher, real publisher article URL (not a news.google.com redirect), publication date, snippet and — optionally — the full article text. Turn on only-new mode and schedule it to get company news alerts that never repeat an article you already received.

It reads the public Google News RSS feed (news.google.com/rss/search), so it is fast (under a second per query for headlines), needs no proxies and no browser at moderate volume, and costs $2 per 1,000 articles.

• 🔎 Any query: keywords, "exact company names", site:techcrunch.com, intitle:, OR, -exclude
• 🌍 Any edition: language (hl), country (gl) and edition (ceid) — e.g. de / DE / DE:de
• ⏱️ Time range: past hour, 24 hours, 7 days, 30 days, year — enforced on every article's date
• 🔗 Publisher URLs resolved from Google's encoded links without a browser
• 📰 Full text via Mozilla Readability — respects robots.txt, skips paywalls gracefully, and you only pay when extraction succeeds
• 🔔 Only new articles across runs (named key-value store), ideal for daily/hourly alerts
• ✅ Publication-date check: compares Google's date with the publisher's own datePublished

What data do you get?

Field	Example
query	site:techcrunch.com startup funding
title	Otter.ai raises $320 million from General Catalyst
publisher / publisherUrl	TechCrunch / https://techcrunch.com
articleUrl	https://techcrunch.com/2026/10/20/otter-ai-raises-320-million-...
urlResolved	true (false = still a Google link)
googleNewsUrl	https://news.google.com/rss/articles/CBMi...
publishedAt	2026-10-20T14:02:11.000Z (UTC)
publisherPublishedAt / publishedAtCheck	2026-10-20T14:01:00.000Z / verified
snippet	Publisher description (when available)
fullText / wordCount / fullTextStatus	Article body / 582 / ok
author, imageUrl, relatedCoverage	When the publisher declares them
language, country, scrapedAt	en-US, US, run time

Each run also writes a summary to the key-value store record OUTPUT: per-status counts, URL-resolution stats, HTTP stats, query errors and the billing total.

Examples

1. Company news alerts (schedule daily, pay only for new articles)

{ "queries": ["\"OpenAI\"", "\"Anthropic\"", "\"Mistral AI\""], "timeRange": "1d", "onlyNew": true, "maxArticlesPerQuery": 50 }

2. Keyword monitoring with full article text for LLM summarisation / RAG

{ "queries": ["AI regulation EU", "chip export controls"], "timeRange": "7d", "fetchFullText": true, "maxArticlesPerQuery": 20 }

3. One publication only (site: operator)

{ "queries": ["site:techcrunch.com startup funding"], "timeRange": "7d" }

4. German edition

{ "queries": ["Elektroauto Batterie"], "language": "de", "country": "DE", "timeRange": "30d" }

Pricing

Pay-per-event, platform usage included:

Event	Price	When
article-scraped	$0.002 ($2 / 1,000)	Every article returned in the dataset
article-fulltext	$0.003	Only when the full text was extracted successfully (fullTextStatus = ok)

• 1,000 headlines with resolved URLs: $2.00
• 1,000 articles with full text (70–85% extracted successfully in our tests): ≈ $4.10–$4.55
• Paywalled, robots-blocked or failed extractions are not charged the full-text fee.
• With onlyNew, already-seen articles are skipped before charging.
• The run never exceeds your Max total charge (USD): budget is reserved before each article and full-text fetch, and the run stops cleanly when it is reached.

Use via API

curl -X POST "https://api.apify.com/v2/acts/<username>~google-news-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"queries":["\"Nvidia\""],"timeRange":"1d","maxArticlesPerQuery":20}'

JavaScript:

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('<username>/google-news-scraper').call({ queries: ['"Stripe"'], timeRange: '7d', fetchFullText: true });
const { items } = await client.dataset(run.defaultDatasetId).listItems();

Use from AI agents (MCP)

Add the Apify MCP server to Claude Desktop, Cursor or any MCP client and expose this Actor as a tool:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?actors=<username>/google-news-scraper",
      "headers": { "Authorization": "Bearer <APIFY_TOKEN>" }
    }
  }
}

Then ask: "Find this week's news about Stripe and summarise the three most important articles." The agent calls the Actor with queries, timeRange and fetchFullText, and gets structured JSON back.

How it works

1. Fetches https://news.google.com/rss/search?q=<query> when:<range>&hl=..&gl=..&ceid=...
2. Parses the RSS (no DTD/entity expansion), strips the "– Publisher" suffix from titles, keeps Google's rank.
3. Resolves each Google link to the publisher URL: legacy link IDs decode offline; current opaque IDs are resolved with Google's own public batchexecute call (1 page fetch + 1 batched POST, no browser, no login).
4. Optional full text: checks robots.txt for the publisher, fetches the page once, extracts with Mozilla Readability, reads JSON-LD/meta for datePublished, author and image, and detects paywalls (isAccessibleForFree: false, HTTP 401/402, "subscribe to continue" teasers, known hard-paywall domains).
5. Dedupes within the run (same story across queries) and, with onlyNew, against previous runs.

Limitations (honest)

• ~100 articles per query. Google News RSS returns at most about 100 items per search (often fewer for niche queries). Split broad topics into several narrower queries or use shorter time ranges and schedules.
• Rate limits. Google requests are serialized with a 400 ms gap and retried with back-off on 429/5xx. In local tests (three back-to-back runs, ~50 Google requests each, 5 queries × 8 articles) we saw zero 429s. Heavy bursts from one IP can trigger Google's "unusual traffic" page; the Actor then stops resolving URLs for the rest of the run (articles keep their Google link, urlResolved: false) instead of hammering. We do not use proxies or solve CAPTCHAs. Comfortable volume: a few hundred queries and a few thousand articles per hour per run.
• Operator queries can come back empty. We observed Google intermittently returning an empty feed for queries containing when: / site:. The Actor retries once without the operators and enforces them itself (time window on pubDate, site: on the publisher domain); the run summary counts these as operatorFallbacks.
• URL resolution costs 2 Google requests per article (~120 KB). Disable resolvePublisherUrls for maximum speed if Google links are enough.
• Snippets. Google's RSS has no real article summary for most items; snippet is filled from the publisher's meta description when full text is enabled, otherwise often null.
• Full text is best-effort. In our tests 70–85% of articles extracted (ok). The rest are reported with a reason: paywalled, robots_disallowed (e.g. Reuters, Politico disallow crawlers), blocked (Cloudflare/403), too_short, fetch_failed. JavaScript-only pages are not rendered.
• Dates come from Google's pubDate. publishedAtCheck = mismatch flags pages whose own date differs by more than 36 h (republished or evergreen pages).
• Terms. Google states the RSS feed is intended for personal feed readers. You are responsible for how you use the data and for complying with Google's and publishers' terms and copyright. Store and redistribute full texts only where you have the right to.

FAQ

Is there an official Google News API? No. Google retired its News API; the RSS feed is the public, stable interface, which is what this Actor uses.

Do I need proxies? Not at moderate volume. The Actor runs without proxies by design and paces its requests.

How do news alerts work? Set onlyNew: true and schedule the Actor (e.g. hourly). Seen articles are remembered for 120 days in the named key-value store google-news-scraper-state under a key derived from your queries (or your own stateKey). Only new articles are returned — and charged.

Why is articleUrl sometimes a news.google.com link? Resolution failed or was disabled (urlResolved: false). Full text is only attempted for resolved URLs.

Can I get articles older than a year? Use timeRange: "any" with specific queries; Google still caps each feed at ~100 items.

What languages are supported? Any Google News edition: set language (hl), country (gl) and optionally ceid.

Will I be charged for paywalled articles? Only the $0.002 article event (you still get the headline, URL and date). The $0.003 full-text fee applies only when text was extracted.

How fast is it? Headlines only (URL resolution off): 25 articles in under 1 s. With URL resolution: ~0.8 s per article (Google pacing). With full text, 5 queries × 8 articles took ~40 s end-to-end in our local test.