Google News Scraper: Articles, Publishers & Real Links

Google News Scraper searches Google News by keyword, topic or publisher and returns one clean row per article: headline, publisher, publisher domain, publish date in ISO 8601, and the rank it held for your query. Every Google search operator works inside a query, and 91 country/language editions are selectable as a single value.

Turn on link resolution and url becomes the publisher's own article address instead of a 250-character redirect token — and urlResolved tells you, per row, whether that succeeded. No API key, no quota, no proxy setup and no browser.

Agent skill: SKILL.md

https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/google-news-scraper.md

What you get

Output field	Meaning
title	The headline. The - Publisher suffix Google appends is removed, since publisher already carries it
url	The publisher's own article URL when resolution is on, otherwise the Google link
googleUrl	Always the Google link, so you can re-resolve later without re-running the search
urlResolved	Whether url is the publisher's. Never leaves you guessing which one you got
publisher	Publisher name as Google files it
publisherUrl	The publisher's home page
publisherDomain	Host of publisherUrl, lowercased and without www. — the field you group by
publishedAt	ISO 8601, UTC
publishedRaw	The feed's own RFC-822 date string, unparsed, for auditing
snippet	Article summary. null unless includeArticleMetadata is on — see Limits
thumbnailUrl	Lead image. null unless includeArticleMetadata is on — see Limits
query	Which input produced the row: the query text, the topic, or the domain
queryType	search, topic or publisher
edition	The edition the row came from, echoed as US:en
position	Rank within its query
scrapedAt	Run timestamp

A RUN_SUMMARY record in the key-value store holds the edition used, every query run and its kind, the resolved date range, the filters applied, requests made, articles saved, duplicates collapsed, how many links resolved and how many did not with the reason for each, how many article pages returned a summary, returned nothing, or could not be read at all, inputs that were rejected and why, and any field that failed its shape check. Failures are named there rather than left to look like a quiet news day.

Input

Field	Type	Default	Notes
queries	array	required	Search terms. Operators work: "quoted phrase", site:reuters.com, intitle:merger, -exclude, OR
topics	array	[]	WORLD, BUSINESS, TECHNOLOGY, ENTERTAINMENT, SPORTS, SCIENCE, HEALTH. Fetched on their own; a section feed takes no search terms, so it is one feed of the last day or two — see Limits
publishers	array	[]	Domains such as reuters.com. Applied on top of each query, so they narrow a search rather than replace it. Given alone, each becomes its own search
edition	string	US:en	Country and language together, from one value. 91 available
maxItemsPerQuery	integer	100	Cap per query, and the main cost control since charging is per article
dateFrom	string		2026-01-15, or relative: 7 days, 36 hours, 3 months
dateTo	string		Same formats. Ignored unless dateFrom is also set
resolvePublisherUrl	boolean	false	Return the publisher's own link. Two extra requests per article, charged separately
includeArticleMetadata	boolean	false	Fetch each article's own summary and image. Switches on resolvePublisherUrl, adds a third request per article, charged separately
deduplicateBy	string	url	url, title or none

At least one of queries, topics or publishers must be non-empty; a run with all three empty fails with a named error rather than returning nothing.

{
 "queries": ["apify OR \"web scraping\""],
 "edition": "GB:en",
 "maxItemsPerQuery": 50,
 "dateFrom": "14 days",
 "resolvePublisherUrl": true
}

Output

One record per article. Every field is present on every record, and null always means the value is not available for that article — never that the column was quietly dropped. Where a null came from an attempt that failed rather than from an article that simply has no such value, RUN_SUMMARY says so and counts it separately.

{
 "title": "OpenAI says it slowed Astra model development over security concerns",
 "url": "https://techcrunch.com/2026/08/07/openai-says-it-slowed-astra-model-development-over-security-concerns/",
 "googleUrl": "https://news.google.com/rss/articles/CBMipgFBVV95cUxNd3B2a2NoaGtOdmJ5WEhERFMwbTZOX3p1MHplbG1ocnRsa25kaDFIRXU5SjVwVmFzRW1ZTWJWZ1dnYmljZ0pMVUZ1SE5RZXpaVy0ydnpjd2lVaHpuTUZEckZDSUI0MGF5N1dYZkItcmNiQkdiVlBPZFZwRENjNzIwYkVQdml5b0Z0c1Rzd3BTMXFWQ2t5N01qQ2w0T1N2bmN4US1Ra2pR?oc=5",
 "urlResolved": true,
 "publisher": "TechCrunch",
 "publisherUrl": "https://techcrunch.com",
 "publisherDomain": "techcrunch.com",
 "publishedAt": "2026-08-07T16:31:00.000Z",
 "publishedRaw": "Fri, 07 Aug 2026 16:31:00 GMT",
 "snippet": null,
 "thumbnailUrl": null,
 "query": "openai",
 "queryType": "search",
 "edition": "US:en",
 "position": 1,
 "scrapedAt": "2026-08-09T01:12:44.000Z"
}

Use cases

• Brand and PR monitoring. Track every mention of a company across 91 national editions, on a schedule, and diff successive runs on googleUrl.
• Competitor press alerts. One query per rival with dateFrom: "1 days" on an hourly schedule, and you see coverage the hour it lands.
• Dated corpus building. For a search query, maxItemsPerQuery above 100 slides the date range into windows automatically, so a year of one subject comes out as one run rather than fifty.
• Retrieval for agents. Current headlines with a real publisher link is exactly the shape a retrieval step wants, and the whole thing is one API call.
• Media landscape analysis. Group by publisherDomain to see which outlets own a story, and compare the same query across editions to see how a story travels.

Limits and behaviour

• snippet and thumbnailUrl are null unless includeArticleMetadata is on. Google News publishes no summary and no image of any kind alongside a headline: not a thumbnail, not a caption, not a first paragraph. The one thing that looks like a summary is a restatement of the headline and the publisher name, which is title and publisher over again. Mapping it through would give you a populated-looking column with nothing in it, so this Actor does not. With includeArticleMetadata on, both fields are read from the article's own page instead.
• A publisher's own summary is often just the headline again. Many outlets set their page description to the title verbatim. The field is best-effort by nature, and stays null when the page declares nothing. If a page could not be read at all — a block, a timeout, a redirect to a consent wall — that is counted apart from "declared nothing" in RUN_SUMMARY, with the reason, so an outage never reads as publisher habit. Neither case is charged.
• Topic feeds are standing sections, so they behave differently from a query. A section takes no search terms, which means the date range cannot travel with the request: it is applied to the articles after the feed is read, and RUN_SUMMARY.dateRange.topicArticlesOutOfRange reports how many that removed. A section also cannot be split into date windows, so a topic returns one feed no matter how high maxItemsPerQuery goes, and it reaches back only as far as the section itself carries — usually a day or two.
• Roughly 100 articles per query per request. Above that the run splits your date range into windows and requests each one, newest first; RUN_SUMMARY.perQuery[...].windows reports how many it used. A larger cap therefore costs proportionally more time, and a narrow date range cannot be split beyond one window per day.
• Resolution is best-effort and says so. Every article that resolves gets the publisher's URL and urlResolved: true. Every article that does not still ships, keeps the Google link in url, and is not charged for the failed attempt.
• Google News indexes headlines, not article bodies. There is no full text here, and no way to search inside one.
• Syndicated duplicates are a property of the source. A wire story runs at a dozen outlets under near-identical headlines. deduplicateBy: "url" keeps them all; "title" collapses them, at the cost of genuinely distinct regional rewrites.
• An edition changes which articles rank, not only their language. The same query in GB:en and AU:en overlaps around 60%, and CA:fr against FR:fr around 20%. Country and language are set from one value so they cannot be made to contradict each other.
• Topic feeds take no search terms. They are standing sections, so topics is fetched separately and is not combined with queries or publishers.
• A bad input never ends the run. An unrecognised topic or a publisher that is not a domain is recorded in RUN_SUMMARY.rejectedInputs and the rest proceeds. The run fails only if everything failed.
• Public data only. No credentials, and nothing that works around access controls.

FAQ

Why do the links look like news.google.com/rss/articles/…? That is how Google publishes them, and it is what every article link is until it is resolved. Set resolvePublisherUrl: true and url becomes the publisher's own address; googleUrl always keeps the original either way.

Why is snippet empty? Because the news feed has no summary field. Turn on includeArticleMetadata and it is read from the article's own page. See Limits.

Do search operators work? Yes — quoted phrases, site:, intitle:, -exclusion and OR, exactly as you would type them into Google News. publishers is a convenience on top of site:, so the two compose.

Why does changing the edition change my results? Each edition is a different national front page with its own publisher mix and ranking, not a translation layer. Pick the edition your audience reads.

Can I get more than 100 articles for one query? Yes. Raise maxItemsPerQuery and the run slides the date range into windows automatically. Give it a dateFrom so it knows how far back to reach.

Do I need a Google API key or a news licence? No. You supply no credentials.

How do I run this on a schedule? Apify schedules, or your own cron against the API. Keep the dataset and diff runs on googleUrl, which is stable per article.

Can I plug it into something else? Yes. Apify API, the client libraries, webhooks, scheduled runs, dataset exports to JSON, CSV or Excel, or MCP. The output is structured JSON.

API example

curl -X POST "https://api.apify.com/v2/acts/arman-bd~google-news-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "queries": ["\"supply chain\" intitle:disruption"],
 "edition": "GB:en",
 "maxItemsPerQuery": 50,
 "dateFrom": "7 days",
 "resolvePublisherUrl": true
 }'

JavaScript example

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('arman-bd/google-news-scraper').call({
 queries: ['apify OR "web scraping"'],
 edition: 'US:en',
 maxItemsPerQuery: 100,
 dateFrom: '14 days',
 resolvePublisherUrl: true,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const a of items) {
 console.log(`${a.publishedAt} ${a.publisher} — ${a.title}`);
 console.log(a.urlResolved ? a.url : `unresolved: ${a.googleUrl}`);
}