Google News Scraper - Real URL & Source Metadata

Google News Scraper — Articles, Real URLs and Source Metadata

This Google News scraper turns any search query into structured JSON rows that carry three things together: the news article from the Google News feed, the real publisher URL decoded from behind the news.google.com redirect, and the destination page's own source identity — canonical URL, Open Graph tags, schema.org publisher and page language. Every response is typed, normalized JSON — no HTML, no selectors, no parsing. After this page you will know exactly which field to join on, what happens when a redirect cannot be decoded, and what each saved row costs you.

---

What is Google News Scraper - Real URL & Source Metadata?

Google News Scraper - Real URL & Source Metadata is an Apify Actor that queries the public Google News RSS search surface and enriches every article with a resolvable, joinable source identity. It returns three related entity types on one flat row: the article (headline, snippet, publish time, GUID), the publisher (name, exact domain, home URL, schema.org publisher name), and the destination page metadata (canonical URL, og:* tags, article:published_time, section, keywords, page language).

No Google account, no login, no cookie, and no Google News API key are required — the Actor reads only public endpoints (src/main.py:910, src/main.py:1134, src/main.py:1169). It is not reachable through an MCP server; you run it from the Apify Console or call it over the Apify API.

• Scrape news articles — headline, cleaned snippet, ISO-8601 publish time, RSS GUID, ranked position
• Scrape the publisher identity — source name, bare publisher domain, publisher home URL, og:site_name, schema.org publisher
• Scrape destination page metadata — canonical URL, Open Graph title/description/image, published + modified time, section, keywords, page language
• Decode the Google redirect into the publisher's own article URL, with an explicit per-row resolveStatus
• Export as JSON, CSV, Excel, XML, HTML or JSONL from the Apify dataset — no proxy management, no HTML parsing on your side

---

What data does this Google News scraper collect?

One run returns four layers of data on a single dataset row: the article as Google News published it, the resolved link that turns a Google redirect into a fetchable URL, the publisher identity, and the destination page's head metadata. Every row also carries the query provenance so a multi-run dataset stays auditable.

Data Type	Key Fields	JSON Field Names
News article	Headline, cleaned snippet, publish time (ISO + relative), feed GUID, result rank	title, snippet, publishedAt, date_utc, date, guid, position, block_position
Resolved article link	Real publisher URL, resolve outcome, Google redirect kept for reference, canonical URL, final URL after redirects	realUrl, resolveStatus, link, rssLink, canonicalUrl, loadedUrl
Publisher / source	Source name from the feed, bare domain, publisher home URL, og:site_name, schema.org publisher	source, publisherDomain, publisherHomeUrl, sourceUrl, domain, siteName, ldPublisherName
Destination page metadata	Open Graph tags, meta description and keywords, article published/modified time, section, page language, fetch outcome	ogTitle, ogDescription, ogImage, image, metaDescription, metaKeywords, articlePublishedTime, articleModifiedTime, articleSection, pageLang, metaStatus
Query provenance	Query you typed, query actually sent to Google, exact feed URL used, optional base64 thumbnail	query, appliedQuery, feedUrl, thumbnail

That is 36 keys, and the Actor writes all 36 on every row — there is no conditional key set to defend against. A field that could not be filled is present with a null value rather than dropped, so CSV exports keep stable columns across runs.

Need more search and discovery data?

These rows join cleanly to other API Empire scrapers on publisherDomain. Pair this Actor with Google Search Autocomplete Scraper (buyer-intent filter) to see which queries surface a topic, Moz Backlink Profile Analyzer or Ahrefs Scraper by Market Category to weight publishers by authority, and Reddit Posts Scraper (buyer-question detection) or Truth Social Keyword Monitor to catch the social reaction to the same story.

---

🔗 How the real publisher URL is resolved

This is the part that separates the Actor from a plain RSS reader, so it is worth reading in full.

The link in the Google News RSS feed looks like https://news.google.com/rss/articles/CBMihgFBVV95cUx.... That token is not a base64-encoded URL you can decode offline — it is an opaque, signed article ID that only Google can expand. The Actor resolves it in two live steps per article (resolve_real_url, src/main.py:1123).

Step 1 — harvest the signature. A plain GET on the news.google.com/rss/articles/<id> URL with a Chrome user agent (src/main.py:775, src/main.py:1134), then three attributes pulled out of the returned HTML with regular expressions (src/main.py:1141):

• data-n-a-id — the internal article id
• data-n-a-ts — the signature timestamp
• data-n-a-sg — the signature itself

It also harvests the Google News web-app build id by matching a 9-digit number inside "…" on the same page (src/main.py:1147). If that harvest finds nothing, it falls back to the pinned constant GOOGLE_BUILD_FALLBACK = '950531894' (src/main.py:786).

Step 2 — call Google's own resolution RPC. The three signature values plus the build id are packed into a garturlreq payload (_build_fbv4je_payload, src/main.py:1048) and POSTed as f.req to:

https://news.google.com/_/DotsSplashUi/data/batchexecute?rpcids=Fbv4je

with source-path set to the article path and a _reqid counter that advances by 100000 per call (src/main.py:1152). The response is Google's chunked RPC envelope; _parse_garturlres (src/main.py:1062) walks the lines, finds the garturlres payload and lifts the publisher URL out of it. If the extracted value starts with http, the row gets resolveStatus: "ok" and the URL lands in realUrl.

What resolveStatus actually means

Value	When it is written
ok	The RPC returned a URL starting with http. realUrl is populated.
blocked	The article page returned a non-200 status, the request raised, the data-n-a-* signature attributes were missing from the HTML, or the batchexecute call answered HTTP 429 or 403 (src/main.py:1136, src/main.py:1144, src/main.py:1170).
failed	The link was not a news.google.com URL, the RPC answered a non-200 that was not 429/403, the POST raised, or no usable URL was found in the response (src/main.py:1128, src/main.py:1173, src/main.py:1182).
skipped	resolveRealUrl was set to false, so no resolution was attempted at all (src/main.py:1339).

What happens when resolution fails

Failure is retried before it is accepted. resolve_with_retries (src/main.py:1109) gives each article up to 3 attempts, and between attempts it escalates a shared proxy rung for the whole run — direct → Apify datacenter proxy → Apify residential proxy (RESOLVE_RUNGS, src/main.py:1081; escalate_resolver, src/main.py:1084). The escalation is shared deliberately: when Google starts throttling the container IP, every worker moves to the next network rung together instead of each one rediscovering the block. Throttled calls are counted and reported in the log at the end of the run (src/main.py:1383).

If all attempts fail, the row is still saved and still charged. This is the single most important billing fact on this page:

• realUrl is null and resolveStatus is blocked or failed
• metaStatus becomes skipped and all 13 destination-metadata fields stay null, because head metadata needs the real URL as its target (src/main.py:1364)
• the row keeps the Google redirect in link and rssLink, plus title, source, publisherDomain, dates and snippet
• Actor.push_data(row, charged_event_name="row_result") runs for that row exactly like any other (src/main.py:1515)

If you would rather not pay for rows without a publisher URL, set skipUnresolved: true. Rows whose resolveStatus is not ok are then dropped before the push, so they are neither saved nor charged (src/main.py:1447), and the run logs how many were dropped (src/main.py:1531).

Fetching the destination page metadata

When fetchSourceMeta is on and a realUrl exists, the Actor fetches only the destination page's <head> — the article body is never extracted (fetch_head_metadata, src/main.py:1281). It tries curl_cffi with a chrome124 TLS fingerprint first and falls back to plain aiohttp if curl_cffi is unavailable or errors (src/main.py:791, src/main.py:1292), parses at most the first 600,000 characters of HTML (src/main.py:1317), and absolutizes a relative canonical against the final URL (src/main.py:1233). For structured data it walks every application/ld+json block looking for a NewsArticle, Article, ReportageNewsArticle, BlogPosting or LiveBlogPosting node, up to 400 nodes deep (src/main.py:1189), and uses it to backfill metaKeywords, articlePublishedTime, articleModifiedTime, articleSection and ldPublisherName.

metaStatus is ok when the head was parsed, blocked when the page could not be fetched or parsed, and skipped when there was no real URL to fetch or fetchSourceMeta was off.

---

Why not build this yourself?

Google does not offer a public Google News articles API — the competing listings say the same thing, with logiover/google-news-scraper describing the official API as "deprecated years ago" and renzomacar/google-news-scraper stating "Google retired its News API years ago" (both checked on the Apify Store, 2026-07-25 — not verified here). So the realistic DIY route is exactly what this Actor does internally, and it has four moving parts to keep alive at once:

1. The signed redirect. The article token cannot be decoded offline. You must fetch the article page, scrape three signature attributes, harvest a rotating 9-digit build id, construct Google's internal Fbv4je RPC payload byte-exactly, and parse a non-JSON chunked envelope. Any change to that payload shape breaks every URL in your pipeline at once.
2. Throttling on cloud IPs. The Actor's escalation logic exists because the batchexecute endpoint behaves differently from a datacenter container IP than from a residential one (src/main.py:1084). A DIY script without a proxy ladder degrades quietly into a dataset of dead Google links.
3. Silent-block detection. A consent page or interstitial arrives with HTTP 200. The Actor treats "not valid XML" or "zero <item> elements" as the block signal instead of trusting the status code (parse_feed, src/main.py:922), then escalates the proxy rung and retries (src/main.py:1013).
4. Publisher-side variance. Every destination site formats its head differently: og:* with twitter:* fallbacks, <link rel="canonical"> in relative form, JSON-LD nested arbitrarily deep — all to be handled without downloading article bodies.

Build it yourself when you need a resolution mechanism no one else controls; use this Actor when you want the rows in your warehouse today with a resolveStatus column you can trust.

---

Why do developers and teams scrape Google News?

Google News is the widest free index of published journalism, but its RSS output alone is close to unusable downstream because the links do not point at publishers. These are the four audiences the field set here is actually built for.

For AI engineers and agent builders

RAG pipelines break on Google redirect links: a fetcher hits news.google.com, receives an interstitial, and indexes nothing. Here you index on realUrl and canonicalUrl, de-duplicate on canonicalUrl so syndicated copies of a story collapse into one document, and use ogTitle + ogDescription + snippet as embedding text when the full body is not licensed for storage. articlePublishedTime and articleSection become vector-store metadata for recency and topic filters, and resolveStatus == "ok" is the gate deciding which rows enter the index at all.

For PR, comms and brand monitoring teams

Schedule the Actor on a brand or executive name with time_period: "last_day" and each row names the outlet three independent ways — source from the feed, publisherDomain from the feed's own <source url> attribute, and ldPublisherName from the article's schema.org block. That triple makes share-of-voice roll-ups honest: group by publisherDomain instead of a free-text publisher name that shifts between "Reuters", "Reuters.com" and "Reuters Staff". realUrl is the link you paste into the client report.

For SEO and content research teams

canonicalUrl tells you which URL the publisher itself considers authoritative — invaluable when a story exists at three paths on one domain. Combine publisherDomain, articleSection and metaKeywords to map which outlets own a topic in Google News, which sections they file it under, and which keyword sets they target. Because appliedQuery and feedUrl are echoed on every row, a dataset assembled from dozens of scheduled runs stays reproducible — you can always see the exact query and feed behind any row.

For researchers, analysts and data teams

Everything returned is public: headlines, publisher names, publish times and public page metadata. Nothing behind a login is touched. For coverage studies, date_utc gives a normalized ISO-8601 timestamp — and stays null rather than being invented when the feed's pubDate cannot be parsed (rss_pubdate_to_iso, src/main.py:861) — while pageLang plus the gl / hl edition pair lets you compare how the same event is covered across country editions. Export to CSV and the column set is stable enough to append month after month.

---

How to scrape Google News (step by step)

1. Open the Actor on Apify. Find Google News Scraper - Real URL & Source Metadata on the Apify Store and click Try for free, or start it from the Apify Console. There is no separate signup, no vendor dashboard and no third-party API key.
2. Enter the required input. The schema requires two fields: query (search term, default "Elon Musk") and maxItems (rows to save, default 10). Google News operators work inside query: "exact phrase", intitle:tesla, site:reuters.com, -excludedword, when:1d, after:2026-01-01 before:2026-01-31.
3. Choose the edition and the enrichment. Pick gl (245 country editions) and hl (135 interface languages), optionally set time_period, and leave resolveRealUrl and fetchSourceMeta on unless you only want raw feed rows. Set skipUnresolved: true to drop rows without a publisher URL instead of saving them.
4. Start the run. Click Start in the Console, or POST to the Apify API run endpoint with your Apify API token. The Actor logs the exact feed URL before it fetches anything (src/main.py:1405), so you can reproduce the query by hand.
5. Download the results. Export the dataset as JSON, JSONL, CSV, Excel, XML or HTML from the Storage tab, or pull it through the Apify API. The default view surfaces all 36 columns with human-readable labels.

What to do when Google News changes its structure

Nothing on your side. The Actor is maintained, and the output schema is the contract: field names, types and the resolveStatus / metaStatus vocabulary stay stable even when the resolution mechanism underneath changes. Scheduled runs, warehouse loaders and agent tools keep working against the same 36 keys.

---

🛡️ How the Actor handles Google News throttling and blocks

Google News answers a lot of requests happily and then, without warning, stops. The Actor treats that as an expected state rather than an error, at three points.

The feed request. fetch_feed (src/main.py:1013) walks a three-rung ladder — no proxy (or your own proxyConfiguration if supplied), then an Apify datacenter proxy group, then an Apify residential proxy pinned to the US. Each rung gets 2 attempts with a 1 s then 2 s backoff and a 45-second request timeout, and every failure is logged with its rung label.

The block signal. HTTP 200 is not treated as success on its own. parse_feed (src/main.py:922) parses the body as XML; a payload that is not valid XML, or that parses but contains zero <item> elements, is logged and treated as blocked, which triggers the next rung. That is what stops a consent interstitial from being reported as "0 articles found".

The URL resolver. Its rung state is shared across all concurrent workers behind an asyncio.Lock, so two workers cannot double-escalate (src/main.py:1084). A 429 or 403 from batchexecute increments a throttle counter and marks that article blocked; the next retry runs on the escalated rung, and the log reports the throttled-call total and the rung the resolver finished on (src/main.py:1383, src/main.py:1392).

Concurrency is bounded by an asyncio.Semaphore from concurrency (src/main.py:1338) with the connector capped at max(4, concurrency * 2) sockets (src/main.py:1350). Proxies stay optional: proxyConfiguration becomes a proxy URL only when useApifyProxy is true (src/main.py:800).

---

⬇️ Input

Every parameter below comes from .actor/actor.json. Two are required by the schema — maxItems and query.

Parameter	Required	Type	Description	Example Value
maxItems	Yes	integer	How many article rows to save. minimum: 1, maximum: 50000, default: 10. Each saved row is one charged result. A value below 1 is raised to 1 in code (src/main.py:1552); a non-numeric value falls back to 10 (src/main.py:1549).	100
query	Yes	string	Search term. editor: "textfield", default: "Elon Musk". Google News operators are supported. An empty or whitespace-only value ends the run cleanly with a warning and saves nothing (src/main.py:1541).	"\"battery recycling\" -tesla"
gl	No	string	Google News country edition, chosen from an enum of 245 country names ("United States", "United Kingdom", "Germany", …). No schema default; the code uses "United States" when unset (src/main.py:1554) and maps the name to an ISO code, falling back to us for anything unmapped (src/main.py:457). Honored by the feed.	"United Kingdom"
hl	No	string	Google News interface language, enum of 135 language names. No schema default; the code uses "English" when unset (src/main.py:1555) and falls back to code en for anything unmapped (src/main.py:463). Honored by the feed.	"German"
lr	No	string	No-op, kept for input compatibility. Enum of 43 language names, mapped to lang_xx and appended to the feed URL, but the Google News RSS surface ignores it. Supplying it triggers a warning in the run log (src/main.py:1407).	"English"
cr	No	string	No-op, kept for input compatibility. Enum of the same 245 country names, appended as &cr=countryXX, ignored by the feed. Also triggers the compatibility warning.	"United Kingdom"
time_period	No	string	Restrict results by age. Enum: last_hour, last_day, last_week, last_month, last_year, custom. No default. Translated into a query operator, not a tbs parameter: when:1h, when:1d, when:7d, when:30d, when:365d respectively (src/main.py:876).	"last_day"
time_period_min	No	string	Custom range start. editor: "textfield", pattern: ^\d{2}/\d{2}/\d{4}$ (MM/DD/YYYY). Used only when time_period is custom; becomes after:YYYY-MM-DD. An unparseable value is dropped from the query without failing the run (src/main.py:900).	"01/01/2026"
time_period_max	No	string	Custom range end. Same editor and pattern; becomes before:YYYY-MM-DD, with the same silent-drop behaviour on an unparseable value.	"01/31/2026"
nfpr	No	integer	No-op, kept for input compatibility. minimum: 0, maximum: 1, default: 0. When 1, &nfpr=1 is appended to the feed URL, but Google News RSS ignores it.	0
filter	No	integer	No-op, kept for input compatibility. minimum: 0, maximum: 1, default: 1. Appended as &filter=0 or &filter=1, ignored by the feed.	1
proxyConfiguration	No	object	Apify proxy settings. editor: "proxy", prefill: { "useApifyProxy": false }. Optional — the run starts direct and escalates to an Apify datacenter and then residential proxy by itself if Google throttles it. Only used when useApifyProxy is true (src/main.py:800).	{ "useApifyProxy": true }
resolveRealUrl	No	boolean	default: true. Decode every news.google.com/rss/articles redirect into the publisher's article URL. Costs two lightweight requests per article. When false, resolveStatus is skipped and realUrl is null on every row.	true
fetchSourceMeta	No	boolean	default: true. Read the destination page's <head> only — canonical, og:*, article:*, keywords, site name, page language, schema.org publisher. The article body is never downloaded. Needs the real URL, so setting this true with resolveRealUrl: false makes the Actor log a warning and turn resolveRealUrl back on for the run (src/main.py:1583).	true
skipUnresolved	No	boolean	default: false. Drop rows whose redirect could not be decoded, so every saved row carries a publisher URL. Dropped rows are not saved and not charged (src/main.py:1447).	false
concurrency	No	integer	How many articles to resolve and enrich in parallel. minimum: 1, maximum: 16, default: 8. Clamped in code to the same 1–16 range (src/main.py:1580).	8
includeThumbnailBase64	No	boolean	default: false. Re-fetches the article page, picks an image and inlines it as a base64 data: URL in thumbnail. Costs an extra page fetch plus an image download per article and runs sequentially, not in parallel (src/main.py:1457). ogImage and image already give you the image URL for free.	false

Example input

{
  "query": "\"battery recycling\" -tesla",
  "maxItems": 100,
  "gl": "United Kingdom",
  "hl": "English",
  "lr": "English",
  "cr": "United Kingdom",
  "time_period": "custom",
  "time_period_min": "01/01/2026",
  "time_period_max": "01/31/2026",
  "nfpr": 0,
  "filter": 1,
  "proxyConfiguration": { "useApifyProxy": false },
  "resolveRealUrl": true,
  "fetchSourceMeta": true,
  "skipUnresolved": false,
  "concurrency": 8,
  "includeThumbnailBase64": false
}

With that input the Actor builds appliedQuery as "battery recycling" -tesla after:2026-01-01 before:2026-01-31 and echoes it, plus the full feedUrl, on every row.

Most common input mistake: setting maxItems to a large number and expecting a large dataset. Google News RSS has no pagination — one query returns roughly 100–110 items and that is the ceiling for that query. The Actor logs this explicitly when maxItems exceeds what the feed returned (src/main.py:1423). The second most common mistake is expecting lr, cr, nfpr or filter to change the results; they are accepted and forwarded, but the RSS surface ignores them, which is why their titles carry "(no-op)".

---

⬆️ Output

Every row is typed, normalized JSON with a fixed 36-key shape. Keys are never omitted — a value that could not be found is null, so CSV and Excel exports keep stable columns. Export formats available from the Apify dataset: JSON, JSONL, CSV, Excel, XML and HTML, plus direct retrieval through the Apify API.

Scraped article — fully resolved row

{
  "position": 1,
  "title": "EU battery recycling rules tighten as gigafactory scrap piles up",
  "link": "https://news.google.com/rss/articles/CBMihgFBVV95cUxOc2pIRk1TQ2xf...?oc=5",
  "domain": "reuters.com",
  "source": "Reuters",
  "date": "6 hours ago",
  "date_utc": "2026-07-25T06:12:00+00:00",
  "snippet": "New end-of-life targets force cell makers to document recovered lithium, cobalt and nickel by mass...",
  "thumbnail": "",
  "block_position": 1,
  "query": "\"battery recycling\" -tesla",
  "guid": "CBMihgFBVV95cUxOc2pIRk1TQ2xf",
  "sourceUrl": "https://www.reuters.com",
  "publishedAt": "2026-07-25T06:12:00+00:00",
  "rssLink": "https://news.google.com/rss/articles/CBMihgFBVV95cUxOc2pIRk1TQ2xf...?oc=5",
  "loadedUrl": "https://www.reuters.com/sustainability/eu-battery-recycling-rules-2026-07-25/",
  "image": "https://www.reuters.com/resizer/v2/ABCDEF.jpg?auth=xyz&width=1200",
  "realUrl": "https://www.reuters.com/sustainability/eu-battery-recycling-rules-2026-07-25/",
  "resolveStatus": "ok",
  "publisherDomain": "reuters.com",
  "publisherHomeUrl": "https://www.reuters.com",
  "canonicalUrl": "https://www.reuters.com/sustainability/eu-battery-recycling-rules-2026-07-25/",
  "ogTitle": "EU battery recycling rules tighten as gigafactory scrap piles up",
  "ogDescription": "New end-of-life targets force cell makers to document recovered lithium, cobalt and nickel by mass.",
  "ogImage": "https://www.reuters.com/resizer/v2/ABCDEF.jpg?auth=xyz&width=1200",
  "metaDescription": "New end-of-life targets force cell makers to document recovered lithium, cobalt and nickel by mass.",
  "metaKeywords": "batteries, recycling, lithium, EU regulation",
  "articlePublishedTime": "2026-07-25T06:12:00Z",
  "articleModifiedTime": "2026-07-25T08:40:11Z",
  "articleSection": "Sustainability",
  "siteName": "Reuters",
  "pageLang": "en",
  "ldPublisherName": "Reuters",
  "metaStatus": "ok",
  "feedUrl": "https://news.google.com/rss/search?q=%22battery%20recycling%22%20-tesla&hl=en&gl=gb&ceid=GB:en&filter=1",
  "appliedQuery": "\"battery recycling\" -tesla"
}

Scraped article — redirect could not be decoded

The key set is identical — all 36 keys are still written. The feed-derived fields (title, source, publisherDomain, publisherHomeUrl, sourceUrl, domain, link, rssLink, guid, dates, snippet, query, appliedQuery, feedUrl, position) are unchanged; the resolution and metadata fields look like this instead. The row is saved and is charged unless skipUnresolved is true.

{
  "realUrl": null,
  "resolveStatus": "blocked",
  "loadedUrl": null,
  "metaStatus": "skipped",
  "canonicalUrl": null, "ogTitle": null, "ogDescription": null, "ogImage": null,
  "image": null, "metaDescription": null, "metaKeywords": null,
  "articlePublishedTime": null, "articleModifiedTime": null, "articleSection": null,
  "siteName": null, "pageLang": null, "ldPublisherName": null
}

Field reference — all 36 keys

Field	Type	How it is produced
position	integer	1-based push order, assigned as rows are saved (src/main.py:1468).
block_position	integer	Same value as position; kept for base-column compatibility.
title	string	RSS <title>, with a trailing - Publisher suffix stripped when present (src/main.py:954).
snippet	string	RSS <description> with HTML tags removed, entities unescaped, truncated to 200 characters plus ... (src/main.py:516).
date	string	Human-relative publish time computed at run time — "6 hours ago", "2 days ago", "Just now". Falls back to the raw pubDate string if it cannot be parsed.
date_utc	string / null	ISO-8601 UTC publish timestamp, or null when the feed's pubDate could not be parsed. Never invented.
publishedAt	string / null	Same ISO-8601 value as date_utc.
guid	string / null	The RSS <guid> — a stable per-article identifier, useful as a de-duplication key.
link	string	The Google News link, taken from the <a href> inside the RSS description when present, otherwise from <link> (src/main.py:963).
rssLink	string / null	The unmodified <link> element from the feed item.
realUrl	string / null	The publisher's own article URL, decoded through Google's Fbv4je RPC. null when resolution did not succeed.
resolveStatus	string	ok, blocked, failed or skipped — see the resolution table above.
loadedUrl	string / null	Final URL after redirects, set only when the destination page was actually fetched.
canonicalUrl	string / null	<link rel="canonical"> from the destination page, absolutized against loadedUrl when relative.
source	string	Publisher name from the RSS <source> element; falls back to the suffix split off the headline (src/main.py:960).
publisherDomain	string / null	Bare host from the feed's <source url="…">, lowercased with www. stripped (normalize_domain, src/main.py:846). The field to group and join on.
publisherHomeUrl	string / null	Publisher home URL exactly as the feed gives it.
sourceUrl	string / null	Same value as publisherHomeUrl; kept for compatibility.
domain	string	publisherDomain when available, otherwise a name-based guess from a small built-in publisher lookup or the URL host (src/main.py:483). See the note below.
siteName	string / null	og:site_name from the destination page.
ldPublisherName	string / null	publisher.name from the page's schema.org NewsArticle / Article JSON-LD.
ogTitle	string / null	og:title, falling back to twitter:title.
ogDescription	string / null	og:description, falling back to twitter:description.
ogImage	string / null	og:image, falling back to twitter:image.
image	string / null	Same value as ogImage; kept for compatibility with the base column set.
metaDescription	string / null	<meta name="description">.
metaKeywords	string / null	<meta name="keywords">, falling back to JSON-LD keywords (lists are joined with commas).
articlePublishedTime	string / null	article:published_time, falling back to JSON-LD datePublished. Publisher-stated, so it can differ from publishedAt.
articleModifiedTime	string / null	article:modified_time, falling back to JSON-LD dateModified.
articleSection	string / null	article:section, falling back to JSON-LD articleSection.
pageLang	string / null	The lang attribute of the destination page's <html> element.
metaStatus	string	ok, blocked or skipped for the destination-page fetch.
query	string	The query exactly as you supplied it.
appliedQuery	string	The query actually sent to Google, including any when: / after: / before: operator added from time_period.
feedUrl	string	The full RSS URL the run fetched — paste it into a browser to reproduce the result set.
thumbnail	string	Base64 data: URL of an article image when includeThumbnailBase64 is true; an empty string otherwise.

Two notes worth carrying into your schema mapping:

• Four pairs are intentional duplicates kept from the base Actor's column contract so existing integrations keep working: position / block_position, date_utc / publishedAt, sourceUrl / publisherHomeUrl, and ogImage / image. Keep one of each pair when you map the output into a warehouse.
• domain is best-effort, publisherDomain is exact. When the feed supplies a <source url="…"> both agree; when it does not, domain falls back to a substring match against a small built-in publisher table, which can mis-assign an outlet whose name happens to contain a shorter publisher's key. Group, join and de-duplicate on publisherDomain or canonicalUrl, and treat domain as a display convenience.

The dataset contains only article rows — the Actor never pushes error, summary or accounting records, so a row count equals an article count.

---

💸 Pricing and what counts as a charged result

This Actor uses Apify's pay-per-event model with a single charged event: row_result, fired once per saved article row (src/main.py:1515). The current rate is on the Actor's Pricing tab; the behaviour is what matters here.

What is charged

• Every row written to the dataset — including rows where resolveStatus is blocked or failed and realUrl is null
• Every row where metaStatus is blocked and the destination-page fields are null

What is not charged

• Rows dropped by skipUnresolved: true — they are skipped before the push, so they are neither saved nor billed
• Runs that end early because query was empty — the Actor exits cleanly having pushed nothing
• Runs where the feed could not be retrieved on any proxy rung — the Actor logs an error and returns zero rows

The Actor also respects the platform charge limit: after each push it checks whether the limit has been reached and stops the loop cleanly with a log line rather than continuing to build rows it cannot bill for (src/main.py:1520).

Getting a resolved-only dataset. The cheapest route is skipUnresolved: true, which prevents unresolved rows from being created at all. If you already have a mixed dataset, filter after export — there are no error rows to exclude, only article rows whose status you check:

# jq — keep only rows that carry a real publisher URL
jq '[.[] | select(.resolveStatus == "ok")]' dataset.json > resolved.json

# Python — the same two filters over an Apify dataset
resolved = [row for row in items if row["resolveStatus"] == "ok"]
enriched = [row for row in resolved if row["metaStatus"] == "ok"]

---

How many results can you scrape with this Google News scraper?

maxItems accepts 1 to 50000 and defaults to 10, but the real ceiling is Google's, not the Actor's: the Google News RSS search surface has no pagination and returns roughly 100–110 items per query. Setting maxItems: 5000 on a single query will not produce 5000 rows, and the Actor logs exactly that when the feed returns fewer items than you asked for (src/main.py:1423).

There is no internal pagination loop, by design. parse_feed truncates the feed's items to maxItems (src/main.py:941) and the push loop breaks once len(pushed) reaches maxItems (src/main.py:1524), so maxItems acts as a cap — never as a target the Actor chases with extra requests you would pay for.

To build a larger corpus, run more queries rather than a bigger maxItems:

• Split by time window — the same query with time_period: "last_day" on a daily schedule accumulates far past the single-feed ceiling, and appliedQuery keeps the slices distinguishable
• Split by operator — site:reuters.com battery recycling, intitle:battery recycling and quoted-phrase variants each return their own feed
• Split by edition — different gl / hl pairs return different articles, and feedUrl records which edition produced each row
• De-duplicate the union on canonicalUrl, falling back to guid

---

How does this Google News scraper compare to other Google News scrapers?

Feature	Google News Scraper - Real URL & Source Metadata	Generic Google News scraper
Real publisher URL	Decoded per article through Google's Fbv4je RPC, with an explicit resolveStatus of ok / blocked / failed / skipped on every row	Typically a nullable URL or a boolean flag: renzomacar/google-news-scraper lists urlResolved (boolean), logiover/google-news-scraper lists originalUrl, scrapeify/google-news-scraper lists articleUrl as resolved "when redirect parameters are present" — as listed on the Apify Store, checked 2026-07-25, not measured here
Destination page metadata	13 fields read from the publisher's own <head>: canonicalUrl, ogTitle, ogDescription, ogImage, metaDescription, metaKeywords, articlePublishedTime, articleModifiedTime, articleSection, siteName, pageLang, ldPublisherName, loadedUrl	None of the three listings above documents a canonical URL, an article section, a page language or a schema.org publisher field (Apify Store, 2026-07-25)
Fields per row	36 keys, always present, null when unfilled	8 documented output fields on scrapeify/google-news-scraper, 15 on renzomacar/google-news-scraper, 17 on logiover/google-news-scraper — counted from their own field tables (Apify Store, 2026-07-25)
Failure transparency	resolveStatus and metaStatus on every row, plus skipUnresolved to drop unresolved rows before they are saved or charged	renzomacar/google-news-scraper documents a urlResolved flag; neither of the other two listings documents a per-row status field for destination-page enrichment (2026-07-25)
Query reproducibility	query, appliedQuery and the full feedUrl echoed on every row	scrapeify/google-news-scraper echoes keyword; logiover/google-news-scraper echoes query, language, country and timeWindow; neither documents the exact feed URL used (2026-07-25)

If you are building an AI agent or a RAG pipeline, the output-format row is the decision-maker — parsing HTML inside an agent loop is a reliability failure mode, not a feature. In this specific domain the second decision-maker is the failure row: a scraper that cannot tell you which URLs it failed to resolve forces you to re-crawl the whole dataset to find out.

---

Integrate this Google News scraper and automate your workflow

This Actor works with any language or tool that can send an HTTP request, because it is invoked exactly like any other Apify Actor — from the Console, from the Apify API, or from a schedule.

REST API integration

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("<YOUR_USERNAME>/google-news-scraper-real-url-source-metadata").call(
    run_input={
        "query": "\"battery recycling\" -tesla",
        "maxItems": 100,
        "gl": "United Kingdom",
        "hl": "English",
        "time_period": "last_day",
        "resolveRealUrl": True,
        "fetchSourceMeta": True,
        "skipUnresolved": True,
    }
)

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row["resolveStatus"], row["publisherDomain"], row["realUrl"])

Authentication is your Apify API token, passed to the client — or as ?token= / an Authorization: Bearer header on raw HTTP calls. Works in Python, Node.js, Go, Ruby and cURL — the run input JSON is identical in all of them.

Automation platforms (n8n, Make, LangChain)

n8n — use the Apify node with the Run an Actor operation, point it at this Actor, and paste the run input JSON. Chain a Get dataset items operation after it and the 36-key rows arrive as n8n items you can filter on resolveStatus before writing to Airtable, Postgres or Slack.

Make — the Apify app provides Run an Actor and Watch Dataset Items modules. A common pattern is a scheduled Run an Actor on a brand query, then an iterator that forwards only rows where resolveStatus equals ok, mapping realUrl and canonicalUrl into a Google Sheets row or a CRM record.

LangChain — wrap the run with ApifyDatasetLoader (or ApifyWrapper.call_actor) and supply a dataset_mapping_function that builds each Document from ogTitle + ogDescription + snippet, with realUrl, canonicalUrl, publisherDomain, articlePublishedTime and articleSection as metadata. Because realUrl is a fetchable publisher link, the same documents can be re-hydrated with full text later.

Apify Schedules and webhooks — schedule the Actor hourly or daily for continuous media monitoring, and attach a webhook on run success to push new rows into your own service.

---

Is it legal to scrape Google News?

Scraping publicly accessible Google News search results is generally lawful in most jurisdictions, but how you use the collected data is a separate question. This Actor requests only public endpoints — the Google News RSS search feed, the public article page, Google's own public URL-resolution RPC, and the publisher's public article page — with no login, no cookie, no paywall bypass and no article body extraction.

What it returns is business and editorial metadata: headlines, publisher names, domains, publish timestamps and public page metadata. It is not a personal-data product; the only personal names that appear are those inside public headlines and public page metadata. Article text remains copyrighted by the publisher, and republishing headlines or snippets verbatim at scale is a copyright and terms-of-service question rather than a scraping one. Respect each publisher's terms and Google's terms of service, and treat the data as a pointer to the source rather than a replacement for it.

Consult legal counsel for commercial use cases involving bulk redistribution of publisher content.

---

❓ Frequently asked questions

Does this Google News scraper work without a Google account or API key?

Yes. No Google account, no login, no cookie and no Google News API key are needed — Google does not offer a public Google News articles API to key into. The Actor calls only public endpoints, and the only credential involved is your Apify API token, and only if you start runs over the Apify API instead of the Console.

Does it return the real article URL or the Google redirect?

Both. realUrl holds the publisher's own article URL, decoded from the signed news.google.com token through Google's Fbv4je resolution RPC, while link and rssLink keep the original Google redirect for reference. canonicalUrl adds the publisher's own declared canonical, which is often the better de-duplication key when a story exists at several paths on one domain.

What happens when an article's redirect cannot be decoded?

The row is still saved — and still charged — with realUrl: null, resolveStatus set to blocked or failed, metaStatus: "skipped" and all 13 destination-metadata fields null. You keep the headline, publisher name, publisherDomain, dates, snippet and the Google link. Each article gets up to 3 resolution attempts with a shared proxy escalation between them before it is given up on. Set skipUnresolved: true if you want those rows dropped instead: they are then neither saved nor charged.

How fresh is the data? Is anything cached?

Every run fetches live. The feed is requested at run time, each redirect is resolved at run time, and each destination page's head is fetched at run time — nothing is cached or replayed between runs. date is computed against the moment of the run, which is why it reads as "6 hours ago"; publishedAt and date_utc give you the absolute ISO-8601 timestamp instead.

Why do lr, cr, nfpr and filter not change my results?

Because the Google News RSS surface ignores them. They are kept in the schema for input compatibility with the base Google-search-style input, and the Actor does append them to the feed URL, but they do not alter the result set — which is why their titles carry "(no-op)". Supplying any of them writes a warning to the run log. Use gl (country edition) and hl (interface language) instead; both are honored by the feed.

Can I scrape paywalled or login-gated news content?

No. Only publicly accessible content is returned, and the Actor never downloads article bodies — it reads the destination page's <head> only. On a hard-paywalled site the head metadata is usually still public, so you typically still get canonicalUrl, ogTitle and ogDescription, and metaStatus tells you whether even that much was reachable. If the publisher blocks the fetch entirely, metaStatus is blocked and the metadata fields stay null while the article row itself is unaffected.

Does it work for AI agent workflows and LLM pipelines?

Yes. There is no MCP server for this Actor, but it is callable as a plain HTTP endpoint by any agent framework — LangChain, LlamaIndex or a custom tool loop — through the Apify API or the Apify client libraries. Every row is typed JSON with stable field names, the same 36 keys every time, null instead of missing keys, HTML stripped out of snippet and ISO-8601 timestamps in publishedAt and date_utc, so there is no parsing step before an LLM sees it. For retrieval, gate on resolveStatus == "ok", embed ogTitle + ogDescription + snippet, and store realUrl, canonicalUrl, publisherDomain and articlePublishedTime as metadata.

How does the Actor handle Google's rate limiting and blocks?

With a three-rung network ladder and honest block detection: direct (or your own proxy), then an Apify datacenter proxy, then an Apify residential proxy, two attempts per rung. A 200 response that is not valid XML, or that carries zero <item> elements, is treated as a block rather than an empty result, and the URL resolver escalates the whole run's rung when batchexecute answers 429 or 403. Throttle counts and the final rung are reported in the run log.

Do I need to configure proxies?

No. proxyConfiguration is optional and the run starts with no proxy at all; the Actor escalates to Apify Proxy on its own if Google throttles it. Supply your own proxyConfiguration with useApifyProxy: true only when you want a specific group or country from the first request onward.

How does it compare to other Google News scrapers on the Store?

The observable difference is the source-identity layer. The three closest listings, checked on the Apify Store on 2026-07-25 — scrapeify/google-news-scraper, logiover/google-news-scraper and renzomacar/google-news-scraper — all advertise a resolved publisher URL, with 8, 17 and 15 documented output fields respectively in their own README field tables. None of the three documents a canonical URL, an article section, a page language or a schema.org publisher field, and none documents a per-row status vocabulary for destination-page fetching. This Actor returns 36 keys per row including those 13 destination-page fields, plus resolveStatus and metaStatus so you can see exactly which rows are complete.

What happens when Google News changes its structure?

The Actor is maintained and the output schema is the contract — field names, types and the resolveStatus / metaStatus vocabulary stay stable on your side even when the resolution mechanism underneath has to be updated. Your scheduled runs, dataset loaders and agent tools keep working against the same keys.

---

Your feedback

Found a bug, hit a publisher whose metadata does not parse, or need a field that is not on the row yet? We want to know. Open an issue on the Issues tab of this Actor's page on the Apify Store — that is the fastest route to the maintainer, and concrete reports (the query, the feedUrl from the row, and the row that came back wrong) get fixed first.