Google News Scraper — Articles & Real URLs

Google News as a flat table, with the links that actually work. Give this Actor a search query, a section name, or both — and get one row per article: the headline with Google's publisher suffix stripped off, the publisher, the publish time, the description, the other coverage Google clustered with it, and the publisher's real URL, resolved from the news.google.com/rss/articles/CBMi… redirect that every other tool hands you.

No API key. No login. No Google account. $0.50 per 1,000 articles, real URLs and proxy included, queries that matched nothing are free, and a run that returns nothing costs nothing at all.

Try it in 30 seconds

{
  "searchQueries": ["artificial intelligence"],
  "maxItemsPerQuery": 50,
  "decodeUrls": true,
  "language": "en",
  "country": "US"
}

One query, the fifty most relevant articles, each with the publisher's own address in url.

Want a section instead? {"topics": ["TECHNOLOGY", "BUSINESS"]}. Want one publisher's last day? {"searchQueries": ["site:reuters.com when:1d"]}. Want the German edition? {"searchQueries": ["bundesliga"], "language": "de", "country": "DE"}.

The problem this solves

A Google News RSS feed does not contain article URLs. It contains this:

https://news.google.com/rss/articles/CBMilwFBVV95cUxPV0dsVjMyTGpNX2ZBSWhDSlVUSHcz…?oc=5

That is a redirect that only resolves inside a browser session. Fetch it with an HTTP client and you get 200 and a page of JavaScript — no Location: header, nothing to follow. Which is why most Google News scrapers, and every "just parse the RSS" snippet, hand you a column of news.google.com links and leave the real work to you.

This Actor does the real work. For each article it resolves the redirect the way the Google News web app does — a signed request to Google's own batchexecute endpoint — and puts the result in url:

https://www.nytimes.com/2026/09/10/science/tristan-buckmaster-openai-math-navier-stokes.html

The Google link is kept in googleUrl, and urlDecoded tells you which of the two url is. Nothing is ever dropped for a failed decode: that row keeps the Google link and is flagged, not discarded.

It costs two extra requests per article, and the input that controls it (decodeUrls) is honest about that. Turn it off and a 100-article feed is a single request.

What comes back

One article row per story:

{
  "ok": true,
  "rowType": "article",              // "article" | "diagnostic"

  "title": "The Mathematician Crushed Between OpenAI and Anthropic Over a Math Problem",
  "url": "https://www.nytimes.com/2026/09/10/science/tristan-buckmaster-openai-math-navier-stokes.html",
  "googleUrl": "https://news.google.com/rss/articles/CBMilwFBVV95cUxPV0dsVjMyTGpN…?oc=5",
  "urlDecoded": true,                // false = the decode failed, url fell back to googleUrl
                                     // null  = decodeUrls was off, no decode attempted

  "source": "nytimes.com",           // the publisher, as Google names it
  "sourceUrl": "https://www.nytimes.com",
  "publishedAt": "2026-09-11T13:16:00.000Z",
  "description": "The Mathematician Crushed Between OpenAI and Anthropic Over a Math Problem  nytimes.com",

  "relatedLinks": null,              // other coverage of the same story, when Google clustered it
  "articleId": "CBMilwFBVV95cUxPV0dsVjMyTGpN…",   // stable across queries — dedupe on this
  "position": 1,                     // 1-based, in Google's own ranking

  "query": "artificial intelligence", // which input produced this row…
  "topic": null,                      // …exactly one of the two is set
  "language": "en",
  "country": "US",
  "feedUrl": "https://news.google.com/rss/search?q=artificial+intelligence&hl=en-US&gl=US&ceid=US:en",

  "error": null,
  "errorType": null,
  "scrapedAt": "2026-09-11T15:43:47.000Z"
}

A clustered story — which is most of what a topic section returns — carries the rest of the coverage:

{
  "title": "Live updates: America marks the 25th anniversary of 9/11",
  "source": "CNN",
  "relatedLinks": [
    { "title": "In solemn ceremonies, U.S. marks 25 years since the Sept. 11 attacks", "source": "Pittsburgh Post-Gazette", "url": "https://news.google.com/rss/articles/CBMingFBVV95cUxNSzJ6…?oc=5" },
    { "title": "▶️ WATCH LIVE: NYC marks 25 years since 9/11 with ceremony at ground zero", "source": "NBC New York", "url": "…" },
    { "title": "9/11 Live Updates: America Gathers to Remember on 25th Anniversary; Trump Speaks at Pentagon", "source": "nytimes.com", "url": "…" },
    { "title": "Photo Gallery: Eagles host large meal-pack event for 9/11 anniversary", "source": "Philadelphia Eagles", "url": "…" }
  ]
}

Every row carries the same columns, so articles and diagnostics export as one rectangular table.

Use cases

• Media monitoring for a brand, a person or a competitor. A scheduled run over a handful of queries, keyed on articleId, gives you every new mention within the hour — with the real URL already resolved, ready to hand to a fetcher or a summarizer.
• Publisher-specific feeds. site:reuters.com when:1d is a one-request daily digest of one outlet, and it works for any domain Google News indexes — including ones that publish no usable RSS of their own.
• Sentiment and narrative tracking. relatedLinks is Google's own clustering: which outlets covered the same story, under which headlines. That is a comparison you would otherwise have to build.
• Market and regional comparison. The same query in en/US and de/DE returns almost disjoint sets of publishers. Run both and you can see how a story is framed in two markets on the same morning.
• RAG and LLM pipelines. Headline, publisher, timestamp and a real URL per row is the input side of a news pipeline. Fetch url, extract the body, embed it — the part this Actor does is the part that is otherwise fiddly.
• Topic dashboards. topics: ["TECHNOLOGY", "BUSINESS", "SCIENCE"] on a daily schedule is three requests and about 150 rows: a curated front page, as data, for a fraction of a cent.

How it works, and why it keeps working

Google News has no API. It has an RSS surface that its own readers consume, at four addresses, all taking the same locale triple.

Source	What it reads	Shape
Search	news.google.com/rss/search?q=<query>&hl=…	RSS 2.0. Full Google News search syntax, ~100 items
Top stories	news.google.com/rss?hl=…	RSS 2.0. The front page, ~38 items, mostly clustered
Topic sections	news.google.com/rss/headlines/section/topic/<NAME>?hl=…	RSS 2.0. 302s to /rss/topics/<id>; redirects are followed
Topic by id	news.google.com/rss/topics/<id>?hl=…	The same, for a section you found in a news.google.com/topics/… link
URL decode	news.google.com/rss/articles/<id> then POST /_/DotsSplashUi/data/batchexecute	HTML, then a )]}'-prefixed envelope carrying the real URL

Four things catch naive scrapers here, and all four are handled:

The links are redirects, not URLs. Covered above — this is the main event.

Headlines carry the publisher glued on. Google writes <title>Headline - Source</title>. Splitting on the last " - " is the usual fix, and it truncates any headline that ends in a dashed clause. This Actor strips the suffix by matching it against the publisher name in <source> — across all 467 items of the seven captured feeds the two agree exactly, every time — and only falls back to the dash when there is no <source> to check against. A German headline in the captures reads Liveticker | 1. FC Nürnberg - Hannover 96 : | 5. Spieltag | 2. Bundesliga 2026/27 - Kicker, and it comes through whole.

The descriptions are HTML inside XML, encoded twice. <a href=…> decodes to HTML, and that HTML decodes again to text. A clustered story's description is an <ol> of five headlines with publishers in <font> tags — parsed into relatedLinks and rendered to plain text in description.

A locale that does not exist answers with a different one. hl, gl and ceid together select one of Google News' regional editions. The English editions take hl=en-US, hl=en-GB; every other language takes the bare code with the region in gl and ceid — verified against captures in both shapes. Ask for an edition Google does not publish and it quietly serves a neighbour, which is why the input descriptions name real pairs.

Underneath: Apify datacenter proxy by default, one pinned session per parallel worker. When Google refuses an exit IP — HTTP 429, HTTP 403, an empty body, an HTML consent page where XML belongs — that session is retired and the same request goes out once more from a different address. Retrying on an address that was just refused only deepens the block, so it is never done. If the second address is refused too, the job stops, keeps every row it already delivered, and files one free blocked row saying where it stopped.

How it compares

• You get the article's real URL, not Google's redirect. That is the difference between a dataset you can fetch from and a dataset you have to post-process with a headless browser.
• Search, sections and any regional edition, one schema. A site: query, a topic id pasted out of the address bar and the German front page all produce the same columns.
• Deduplicated within the run, and you are not billed twice. The same story shows up in overlapping queries every day. Rows are deduplicated on articleId before the decode, so a duplicate costs neither the two requests nor the row.
• Failures are free and legible. An empty search, an invalid topic, a block or a run that ran out of time produce a diagnostic row with an errorType you can branch on — and no charge. A run that returns nothing at all finishes FAILED with the reason in its status message, never a green run containing an apology.
• A partial run is kept, not thrown away. Hit maxRunSecs or your charge ceiling halfway through a feed and you keep the rows already written.
• Honest about the cost of the useful part. Decoding is two requests per article and the input says so, in the Console, before you run it.
• No login, ever. No Google account, no cookie jar, no session token. Public endpoints only, which is also why it runs under Apify's limited permissions.

Input reference

Field	Type	Default	What it does
searchQueries	array of strings	prefilled with one	Google News searches. Full search syntax: site:, when:1d, "quoted", OR, -excluded, intitle:
topics	array of strings	[]	TOP, WORLD, NATION, BUSINESS, TECHNOLOGY, ENTERTAINMENT, SPORTS, SCIENCE, HEALTH — or a raw CAAqKggK… topic id, or a news.google.com/topics/… link
language	string	en	Language code for the edition. en, de, fr, es, ja, or a regional tag like pt-BR
country	string	US	Country code for the edition. Pair it with a language actually published there
maxItemsPerQuery	integer	100	Articles kept per feed, in Google's ranking. Caps rows, never requests — there is no paging
decodeUrls	boolean	true	Resolve each article's real publisher URL. Two extra requests per article. Off = one request per query, url = the Google link
maxConcurrency	integer	4	Queries, topics and decodes in parallel. Each worker keeps its own proxy session
maxRunSecs	integer	240	Whole-run wall-clock budget. When it runs out the Actor keeps what it has and files a free diagnostic row for each entry it never reached
proxyConfiguration	object	Apify datacenter	Google answered datacenter addresses on every capture, so the cheap proxy is the default

Output reference

Column	On which rows	What it is
ok, rowType	all	true plus article, or false plus diagnostic
input	all	The entry this row came from, as you supplied it
query, topic	all	Which input produced the row — exactly one of the two is set on an article row
articleId	article	Google's own identifier. Stable across queries, topics and editions — the key to dedupe on between runs
title	article	The headline, with the - Publisher suffix removed
url	article	The publisher's real address when resolved, the Google redirect when not
googleUrl	article	The news.google.com/rss/articles/… redirect, always
urlDecoded	article	true real URL, false decode failed, null decoding was off
source, sourceUrl	article	The publisher's name and home page, from the feed's <source> element
publishedAt	article	When Google says it was published, as UTC ISO 8601
description	article	The feed's description as plain text. A bulleted list for a clustered story. Clipped to 4,000 chars
relatedLinks	article	{title, url, source} for the other coverage Google clustered, up to 20. Their URLs are Google redirects and are not resolved
position	article	1-based position in the feed Google served — Google's ranking at that moment
language, country	all	The Google News edition this row came from
feedUrl	all	The exact address the row was read from, locale parameters and all
error, errorType	diagnostic	What went wrong, and a type to branch on
scrapedAt	all	When the row was written

errorType is one of no-results, invalid-input, blocked, timeout, unavailable.

Pricing

$0.50 per 1,000 articles. Pay-per-event, with the URL decode and its proxy traffic already inside that number — there is no separate proxy line on your bill for this Actor, and no surcharge for decoded rows.

Event	What triggers it	FREE	Starter	Scale	Business
Article returned (primary)	One article row written to your dataset	$0.0005	$0.0005	$0.0004	$0.0003
Run started	Once per run, after the first paid row	$0.001	$0.001	$0.001	$0.001

Worked example. Five queries a day, 50 articles each, URLs decoded, with roughly 20% overlap between them:

• 250 gross − 50 duplicates = 200 billable articles × $0.0005 = $0.10
• 1 run start = $0.001
• 50 duplicates, 0 empty queries = $0.00
• Total: $0.101 per run, about $3.03 a month daily

Topic dashboard example. Three sections, decoding off: ~150 articles × $0.0005 + $0.001 = $0.076 per run, from three requests.

Undecoded bulk example. Twenty queries, 100 articles each, decodeUrls: false: 2,000 articles × $0.0005 + $0.001 = $1.001, from twenty requests and under a minute.

What you are never charged for: a search that matched nothing, a topic that is not a topic, a story you already have from another query, a query the run never reached before maxRunSecs, or a request Google blocked. If a whole run comes back empty it finishes FAILED and bills nothing at all, start fee included.

Set ACTOR_MAX_TOTAL_CHARGE_USD on a run and the Actor stops fetching once the ceiling is in sight, rather than handing you rows it cannot bill or billing you for rows it cannot hand over. It finishes SUCCEEDED with the ceiling named in its status message, and everything already delivered is yours.

Limits, and the ones that might bite

No article text. This Actor returns the headline, Google's one-line description and a URL. It does not fetch the article body — that is a different job, often behind a paywall, and pretending otherwise would be the kind of claim that fails quietly on a third of your rows. url is the address to hand to whatever does fetch it.

Google decides how many articles a feed holds, and there is no paging. The captured feeds range from 38 items (top stories) to 112 (a German search); a typical search returns a little over 100. maxItemsPerQuery trims that list, it does not extend it. To get more coverage, send more queries — narrower ones, or the same one against several editions.

when: is a hard filter, not a hint. when:1d on a quiet query can legitimately return nothing, and you will get a free no-results row rather than yesterday's news.

Decoding costs two requests per article and can fail. It is the slow part of any run: budget roughly a second per article at the default concurrency. Rows whose decode failed keep the Google link and say urlDecoded: false — you can re-run just those, or follow the link by hand. Expect the occasional failure rather than a clean 100%.

relatedLinks URLs are not decoded. A clustered story has four or five of them, so resolving them would multiply the cost of a run by about five for data most users do not fetch. They are Google redirects; feed one back in as its own article if you need it.

Publisher names are Google's, and they are inconsistent. The same outlet appears as nytimes.com in one feed and as a masthead in another. sourceUrl is the stable thing to group on.

Editions are not interchangeable. en/US and de/DE return almost disjoint publisher sets for the same story, and an edition Google does not publish is silently replaced with one it does. Use real language/country pairs.

No historical archive. Google News serves what is current. when:1y reaches back within the index, but a run captures the feed as it stands and there is no way to ask for a specific past day.

The upstream format may change. Google's RSS surface and the batchexecute decode are internal endpoints that change without notice — that is true of every tool that reads this data, including the ones that do not say so. When a shape changes, rows stop arriving and you get free blocked diagnostic rows, or articles with urlDecoded: false, rather than quietly wrong data — and a run that returns nothing bills nothing.

Rate and reliability. Requests go out through proxy sessions pinned per worker, one rotation per block, and a pause between the requests of the same job. Four in parallel is the default; raising maxConcurrency speeds a decoded run up and raises the chance of a block with it.

Use it from an AI agent, or from code

One JSON object in, one flat array out — the shape agent runtimes want. The Actor runs with limited permissions, uses pay-per-event pricing and never enters Standby, so it works over the Apify MCP server and with x402 agentic payments. The Integrations tab pushes results to Slack, a webhook, Zapier, Make, Google Sheets, Snowflake or BigQuery.

curl -X POST "https://api.apify.com/v2/acts/insight.solutions~google-news-api/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"searchQueries":["artificial intelligence"],"maxItemsPerQuery":50,"decodeUrls":true}'

# pip install apify-client
from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("insight.solutions/google-news-api").call(run_input={
    "searchQueries": ["site:reuters.com when:1d"],
    "topics": ["TECHNOLOGY"],
    "maxItemsPerQuery": 50,
    "decodeUrls": True,
})

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row["rowType"] == "article" and row["urlDecoded"]:
        print(row["publishedAt"], row["source"], row["url"])

FAQ

Do I need a Google account or an API key? No. Every source here is public: the RSS feeds any reader can subscribe to, and the same URL-resolution call the Google News web page makes when you click an article.

Why is url sometimes still a news.google.com link? Either decodeUrls was off (urlDecoded is then null), or the decode failed for that article (urlDecoded is false). The row is never dropped over it — the Google link still works in a browser, and you can re-run just those rows.

How do I deduplicate between runs? On articleId, which is Google's own identifier and is stable across queries, topics and editions. An incremental pipeline is an upsert keyed on it. The run already deduplicates within itself.

Can I search one publisher only? Yes: site:reuters.com, site:bbc.co.uk, and so on. Combine it with when:1d for a daily digest of one outlet in a single request.

What is the difference between a topic and a search? A search is Google News' own search, ranked by relevance, and returns about 100 items. A topic is a curated section, ranked editorially, and returns fewer — 38 to 67 in the captures — but nearly all of them are clustered stories with relatedLinks filled in.

Where do I find a raw topic id? Open a section on news.google.com and copy the long CAAqKggK… string out of the address bar, or paste the whole news.google.com/topics/… link — both work. That is how you follow a section narrower than the nine named ones: one league, one city, one company.

Does country change which articles I get? Yes, completely. The edition decides which publishers are indexed at all. The same query in en/US and en/GB returns overlapping but different sets, and de/DE returns a nearly disjoint one.

Why did I get fewer articles than maxItemsPerQuery? Google served fewer. There is no second page to ask for. Check the no-results diagnostic rows if a query returned none at all — a when: clause that is too tight is the usual reason.

What happens if one query fails? The others still run. The failed one produces a free diagnostic row and the run finishes SUCCEEDED. If every entry fails, the run finishes FAILED and you are billed nothing at all.

Is the data fresh? Live. Every run reads Google News at that moment; nothing is cached between runs. Rankings move by the hour, which is why every row carries scrapedAt.

Legal and data-protection notes

• Public endpoints only. A Google News RSS feed is published precisely so that software will read it, and the URL-resolution call is the one the Google News page itself makes. The Actor never logs in, never accepts cookies or session tokens, and never takes an API key belonging to anyone else.
• Headlines and descriptions belong to their publishers. Titles and the one-line descriptions are short extracts of copyrighted works. Indexing, analysis, monitoring and linking are the ordinary uses and are what this is built for; republishing them as your own content is your call and your responsibility.
• url is a link, not a licence. It is the address a reader would open. Fetching article bodies at scale is a cost to the publisher and may breach their terms or their robots policy; this Actor never fetches them.
• News is about people. Rows can name private individuals and can carry data that is sensitive in your jurisdiction. Have a lawful basis before you store or profile it, and keep in mind that a news article is a snapshot that publishers sometimes correct or withdraw.
• Not affiliated with Google LLC or with any publisher whose headlines you retrieve. All names and trademarks belong to their owners and are used only to describe which public endpoints this Actor reads.

Our other Actors

Every Insight Solutions Actor is pay-per-result with no browser, no login and no API key, and every one of them returns free diagnostic rows instead of billing for failures. Prices are per 1,000 results.

Video, audio & social

• YouTube Transcript API — captions as timed segments, text, SRT or VTT, with language fallback and translation.
• YouTube Comments API — comments and replies with likes, pinned and hearted flags, newest or top sort.
• YouTube Channel API — a channel's videos, Shorts and live streams, plus YouTube search.
• Podcast Search, Episodes & Charts API — Apple Podcasts search, charts and full episode feeds.
• Bluesky Scraper — profiles, posts, followers and follows from the public AT Protocol API.
• Telegram Channel Scraper — posts, views and channel stats from public Telegram channels.
• Substack Scraper — posts with full free text, comments and publication profiles.

News, documents & the web

• Website to Markdown — Content Extractor for LLMs & RAG — any site as clean Markdown, text and heading-aware chunks.
• Internet Archive API — archive.org search, item metadata, files and reviews.
• Wayback Machine Toolkit — archived URL inventories, snapshots and text diffs between dates.
• Website Technology Detector — the tech stack behind any site, with the evidence for each detection.
• Domain Intelligence API — DNS, RDAP registration, TLS certificate and HTTP facts in one row per domain.
• SEO Page Audit — sitemap crawl with on-page checks, structured data and broken-link reports.
• Keyword Suggestions API — Google, YouTube, Bing, Amazon and eBay autocomplete with alphabet and question expansions.
• Website Contact Extractor — emails, phone numbers and social profiles from any list of websites.

Business, finance & jobs

• Congress & Insider Trades API — STOCK Act periodic transaction reports and SEC Form 4 insider trades in one schema.
• SEC EDGAR API — filings, XBRL financials and full-text search by ticker or CIK.
• Y Combinator Companies, Batches & Founders — the YC directory with founders and social links, filterable by batch, industry and hiring status.
• Career Site Jobs API — jobs straight from Greenhouse, Lever, Ashby, Workable and 10+ other ATS career sites.
• New Job Postings Monitor — new, closed and changed postings on the career sites you watch.
• Shopify Products API — any Shopify store's catalogue, variants, prices and stock signals.

Apps & games

• App Store & Google Play Reviews API — reviews from both stores with ratings, versions and developer replies.
• App Store Top Charts & App Search API — Apple top charts by country and genre, plus app search and details.
• Steam Reviews API — Steam reviews with playtime, helpfulness and game details.
• Steam Game Data API — prices, tags, review scores, live player counts and top charts.