Google News Scraper: Search, Topics, Decoded URLs

Google News Scraper: Search, Topics, Decoded URLs

Scrape Google News without an API key or a login. Search with Google's own operators, read topic, place, publisher and top headline feeds in any country edition, get the real publisher URL instead of the Google redirect link, and go past Google's 100 article limit by splitting your date range day by day.

Pure HTTP, no browser, no proxy. A run of 20 articles with decoded URLs finishes in a few seconds on 256 MB. You pay per new article, and a named memory means a daily monitor never pays for the same article twice.

Quick start

1. Put your search terms in Search terms, topics, places or Google News URLs, one per line. Operators work: after:2026-09-01, before:2026-09-30, when:7d, site:bbc.com, source:Reuters, "exact phrase", -excluded. You can also paste a news.google.com search, topic, section or place URL.
2. Choose the Feed type. Search is the default. Topic takes TECHNOLOGY, BUSINESS, SPORTS and the rest, or a topic id. Place takes a city, region or country name. Publisher takes bbc.com or Reuters.
3. Set Country edition and Language if you do not want the United States English edition.
4. Set Last N days or an explicit Date from and Date to, then leave Deep search on.
5. Set Max articles in total, press Start, and download the dataset as JSON, CSV or Excel.

To turn a run into a monitor, fill in Dedup memory name, save the run as a task and schedule it. Every later run delivers only articles that memory has not seen, and the known ones cost nothing.

What you get

One row per article. The fill rates below were measured on 2026-09-15 and the raw measurement is committed in the repository at research/fill-rates-2026-09-15.json, not rounded up from memory: 300 live articles across three queries and 221 publishers, plus a 60 article topic feed, a 100 article search feed and a 45 article body sample.

Field	What it is	Fill rate
title	Headline, with the trailing publisher name removed	100%
url	The publisher's own article URL when decoding succeeded, otherwise the Google News link	100%
googleUrl	The original news.google.com link, always present	100%
urlDecoded	Whether url is the publisher URL or still the Google link	100% decoded
sourceName	Publisher name as Google News reports it	100%
sourceUrl, sourceDomain	Publisher home page and its host	100%
publishedAt	Publication time, ISO 8601 UTC	100%
snippet	The feed's summary line. Google repeats the headline here for essentially every item, so treat it as a headline, not a summary	100% present, 100% equal to the headline
query, feedType, shardWindow	Which query, feed and date window produced the row	100%
country, language, locale	The Google News edition the row was read from	100%
isNew	False only for articles your dedup memory already knew	100%
relatedCoverage	Other articles Google clustered with this one	54 of 60 rows on a topic feed, 0 of 100 on a search feed
body, wordCount, paywalled, bodyRetrieved, bodyStatus	Publisher article text, only when you ask for it	32 of 45 pages returned a body, median 564 words
bodySourceUrl	The page the text was really read from, after any publisher redirect	present on every row that asked for a body

Every run that completes writes one run-summary row with per query counters: articles found, delivered, filtered, dropped as outside the date range, deduplicated in the run, known and skipped, feed windows requested, windows that came back truncated, feed errors, decode successes and failures, and body successes and failures. It also reports what the run actually charged, split into what was requested and what the platform confirmed.

Going past the 100 article limit

One Google News feed request returns about 100 items, whatever you ask for. There is no page or offset parameter. Deep search splits your date range into one request per day, and splits the still open day into rolling hour windows when that day is still full.

Measured on 2026-09-15 for the query "artificial intelligence": one seven day feed returned 100 articles, seven one day windows returned 694 unique articles. Google News is a live feed, so the exact number moves between runs; repeated measurements the same day ranged from 496 to 696 against an unchanging 100.

When a window still comes back at the limit, the run summary counts it in cappedShards and the Actor says so in plain text. It does not pretend the window was complete.

How much does it cost to scrape Google News?

Pay per event. A run that finds nothing pays only Apify's standard start fee of $0.00005 and no per article charge at all.

Event	Free and Bronze	Silver	Gold and above
Article	$0.001	$0.0009	$0.0008
Article body	$0.002	$0.0018	$0.0016
Actor start	$0.00005	$0.00005	$0.00005

Apify bills the start event once per gigabyte of run memory. This Actor runs on 256 MB by default, so it is one start event per run unless you raise the memory yourself.

An Article is charged once, when a new unique article is delivered. Decoding the publisher URL is included in that price. An Article body is charged only when the publisher's text was actually delivered, at least 100 words. Duplicates, filtered rows, articles your dedup memory already knows, warning rows and the run summary are all free. So are pages that turn out to be blocked, empty, a cookie consent screen, a subscription teaser, or a different story from the one the link promised.

A page the publisher marks as not free to read but still publishes the whole text is delivered, flagged paywalled: true, and charged like any other body: you got the article. A page that only offers a subscription teaser is rejected and costs nothing. Nothing is ever unlocked or bypassed.

So 1,000 new articles with decoded URLs cost $1.00005 on the Bronze tier, or $0.80005 on Gold: the articles plus one start event. Adding article bodies to all 1,000 costs up to $2.00 more, and less in practice, because a page that is blocked, empty, a subscription teaser or a story we cannot identify is not charged. The Actor start fee is billed per gigabyte of memory and this Actor runs on 256 MB, so it is one event per run.

Input

Every field has a plain English title and description in the Console. The ones that change what you pay:

• Max feed requests per query caps how many requests one query may make, hour windows included. Days beyond the cap are not requested, and the run summary counts them.
• Max articles per query and Max articles in total cap the charged rows. The total is shared fairly between your queries, then any unused budget goes back to the queries that still have articles waiting.
• Decode publisher URLs is on by default in the Console. It is included in the Article price.
• Include the article body is off unless you switch it on, including for API callers, so nobody is silently upcharged.
• Dedup memory name makes known articles free. Re-emit known articles adds them back as free rows flagged isNew: false, and those free rows carry feed metadata only. They have their own budget, so a run with the option on can return up to twice Max articles in total rows. The charged part is never more than Max articles in total, whatever the split between new and known turns out to be.
• Only these publishers and Never these publishers accept domains or names. For search terms you type, the include list is pushed into the query so Google filters server side; for a pasted Google News URL, a topic, a place or top headlines, both lists are applied to what comes back. Either way a filtered out article is never charged.

An empty input is valid: the Actor returns the country's top headlines, capped at 20 articles.

The Console prefills and the values an API caller gets when they leave a field out are deliberately different. Omit them from an API call and you get: Decode publisher URLs off, Include the article body off, Max articles per query 100 and Max articles in total 1,000. Nothing that costs extra is ever switched on for a caller who did not ask for it.

What this Actor does NOT do

• It does not bypass paywalls or consent walls. A page it cannot read is delivered with

  bodyRetrieved: false
  and a bodyStatus that says why, and it is not charged.
• It does not scrape the Google News web app or the Google Search news tab. Feeds come from the public RSS endpoints; decoding a link additionally loads that article's own Google News page and asks Google's link resolver for the publisher URL; article bodies come from the publisher's page.
• Google News publication pages (news.google.com/publications/...) and Full coverage story pages (/stories/...) have no RSS feed. Pasting one returns a free warning row telling you to use the publisher domain instead. Publisher filtering by domain or name works.
• Google's after: and before: operators accept dates, not times, so a past day cannot be split into hours. Only the day that is still open can, using rolling when:Nh windows.
• It does not return article images or author names.
• snippet is the feed's own summary line, and Google fills it with the headline again for almost every item. If you need a real summary, take the article body.
• It is not a Google Search scraper, and it does not read the Google Search News tab.
• Two runs of the same dedup memory started at the same time can both deliver the same article. The memory is a key value record, not a lock. Schedule a memory's runs so they do not overlap.
• If the platform refuses a charge, the row is still delivered and the article is remembered, so it is never billed later. The run summary reports what was requested and what the platform confirmed, and they can differ.
• Article bodies are read from wherever the publisher's link ends up after redirects, which is reported in bodySourceUrl. The cookie consent detector is tuned on English wording; on other languages a consent page is usually still rejected for being too short, but not always.
• One feed request can return an article a few hours outside the day it was asked for. Rows outside the requested range are dropped, counted in the summary as outOfWindow, and never charged.
• A query that returns nothing new for twelve windows in a row stops, and a run whose first forty windows return no article at all stops. Both are reported in the run summary. On a very sparse historical range this can end a query before the oldest days, so narrow the query rather than widening the range.
• If the dedup memory you named cannot be opened, the run delivers the articles and charges nothing for them. If it stops saving part way through, everything from that point on is free. Articles delivered and saved before the failure are charged normally, because they are safely remembered. It is not possible to promise "pay once per article" for rows the memory cannot record, so those rows are free instead.
• Article bodies are only taken from a page that agrees it is the article we asked for. A page whose structured data names a different story returns bodyStatus: identity-mismatch; a page whose markup carries several stories the Actor cannot tell apart returns bodyStatus: ambiguous-page. Both deliver no body and cost nothing. Where a publisher labels nothing at all, the single substantial story on the page is used, so this is a strong filter rather than a guarantee.

Use it from the API

curl -X POST "https://api.apify.com/v2/acts/automation_craft~google-news-scraper/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"queries":["artificial intelligence"],"lastNDays":7,"maxTotal":200,"maxArticlesPerQuery":200,"decodeUrls":true}'

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('automation_craft/google-news-scraper').call({
    queries: ['site:techcrunch.com'],
    lastNDays: 30,
    maxTotal: 500,
    maxArticlesPerQuery: 500,
    decodeUrls: true,
    memoryName: 'techcrunch-monitor',
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();

from apify_client import ApifyClient
client = ApifyClient("YOUR_TOKEN")
run = client.actor("automation_craft/google-news-scraper").call(run_input={
    "queries": ["TECHNOLOGY"],
    "feedType": "topic",
    "country": "GB",
    "language": "en-GB",
    "maxTotal": 100,
    "maxArticlesPerQuery": 100,
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item.get("title"), item.get("url"))

FAQ

Can I scrape Google News without the official API?

Yes. Google retired its News API years ago, and the public RSS feeds are what is left. This Actor reads those feeds, so there is no API key, no Google account and no login anywhere in the flow.

How do I get more than 100 articles from Google News?

Leave Deep search on and give the run a date range. One feed request is capped at about 100 items and has no page or offset parameter, so the only way through is narrower windows. The Actor requests one window per day and deduplicates across them. In a live test, seven day windows returned 694 unique articles where a single seven day feed returned 100.

How do I get the real article URL instead of the Google News redirect link?

Leave Decode publisher URLs on. The Actor resolves each Google News link to the publisher's own URL and puts it in url, keeping the original in googleUrl. When Google refuses to resolve a link, the row is still delivered with urlDecoded: false and the Google link, and it is not billed differently.

Can I get the full article text?

Switch on Include the article body. The Actor fetches the publisher's page and extracts the article text, a word count and a paywall flag. In a 45 article sample taken on 2026-09-15, 32 pages returned a body with a median of 564 words; the other 13 were refused by the publisher, failed to connect, offered only a subscription teaser, carried several stories the Actor could not tell apart, or were too short to be worth charging for. None of those were charged. The full breakdown is in research/fill-rates-2026-09-15.json.

How do I search Google News for older articles or a fixed date range?

Use Date from and Date to, or Last N days. On search and publisher feeds they are translated into Google's after: and before: operators, and Deep search then walks the range one day at a time. Topic, place and top headline feeds have no date operators at all, so for those the range is applied to each article's publication time instead, and the run summary says so. Google indexes older articles unevenly, so a range far in the past returns fewer rows per day than a recent one.

Why does this Actor run with limited permissions?

Least privilege. It only needs its own storages, so that is all it asks for. The one shared thing it touches is the cross run dedup memory, which is a named key value store it creates in your account on first use and reads back on later runs.

More data tools by Automation Craft

• Google Trends Scraper - Compare and Trending Now
• Substack Scraper: Posts, Notes and Profiles
• Telegram Channel Scraper: Posts, Views, Dates
• Meta Ad Library Scraper - All Placements, Filters
• G2 Reviews Scraper: Ratings, Pros and Cons
• LinkedIn Jobs Scraper - No Login, Real Dates
• Amazon Product Scraper - Search, Best Sellers

Changelog

1.0 (2026-09-15): first release. Search, topic, section, place, publisher and top headline feeds; pasted Google News URLs; day and hour window sharding past the 100 item limit; publisher URL decoding; optional article bodies with a paywall flag; named cross run dedup memory; per query and per run caps; run summary with per query counters.