Google News Scraper: Full-Text Articles to Markdown

Search Google News by keyword or topic and get every article's full text as clean Markdown, plus RAG-ready chunks.

The Actor resolves each news.google.com/rss/articles/… wrapper to the publisher URL, fetches the page, and extracts the main article with trafilatura. You get structured metadata plus Markdown — not just headlines and links.

It is built for AI/RAG pipelines and pairs with the same developer's document-to-markdown-rag Actor. Chunks use the same shape: text, source, section, chunkIndex, startChar, endChar.

Why this Actor

Headline scrapers such as data_xplorer/google-news-scraper-fast and lhotanova/google-news-scraper are excellent for monitoring titles and dates. They stop at the Google News card (and sometimes a decoded URL or image).

This Actor goes one step further:

	Headline scrapers	This Actor
Title, publisher, date	Yes	Yes
Resolved publisher URL	Often an add-on	Always (when Google still serves the splash)
Full article body	No	Markdown
RAG chunks	No	Heading-aware, with offsets
Paywalled / blocked pages	Usually a failed row	Metadata kept, full-text event not charged
Price	about $0.004 / headline	$0.002 / run + $0.0015 / article + $0.003 only when full text succeeds

Use a headline scraper when you only need the feed. Use this Actor when you want to embed, summarize, or cite the article itself.

What you get

Each dataset item:

{
  "title": "Introducing GPT-6.1 Terra",
  "publisher": "OpenAI",
  "publisherUrl": "https://openai.com",
  "url": "https://openai.com/index/introducing-gpt-6-1-terra/",
  "googleLink": "https://news.google.com/rss/articles/CBMi…",
  "publishedAt": "2026-10-26T18:05:36Z",
  "author": null,
  "language": "en",
  "description": "Introducing GPT-6.1 Terra  OpenAI",
  "markdown": "# Introducing GPT-6.1 Terra\n\n…",
  "wordCount": 842,
  "chunks": [
    {
      "text": "# Introducing GPT-6.1 Terra\n\n…",
      "source": "https://openai.com/index/introducing-gpt-6-1-terra/",
      "section": "Introducing GPT-6.1 Terra",
      "chunkIndex": 0,
      "startChar": 0,
      "endChar": 1580
    }
  ],
  "status": "ok",
  "statusReason": null,
  "fullTextStatus": "extracted",
  "httpStatus": 200,
  "query": "OpenAI",
  "sourceType": "keyword"
}

status is one of:

• ok — full Markdown extracted; the full-text-article event is charged
• metadata_only — paywall, bot challenge, short/empty body, or full text turned off; only the article event is charged
• decode_failed — Google News wrapper could not be resolved; metadata from RSS is still returned

fullTextStatus says what happened when the Actor fetched the full text:

fullTextStatus	Meaning	full-text-article charged?
extracted	Markdown extracted (status: ok)	Yes
blocked	The publisher refused the scraper: HTTP 401, 402, 403, 407, 429, or 451, or a bot-challenge page ("checking your browser", "access denied")	No
paywalled	Subscriber wall	No
fetch_failed	Network error or another 4xx/5xx (httpStatus has the code)	No
too_short	Page loaded but had less usable text than minFullTextChars	No
failed	The extractor crashed on this page	No
skipped_budget	Your max charge for the run was reached, so the body was withheld	No
disabled	extractFullText is off	No
not_attempted	The Google News link could not be decoded	No

httpStatus is the HTTP status code the publisher returned. Only a row with fullTextStatus: "extracted" and non-empty markdown is billed for full text. The charging code checks this again just before it bills. A blocked row still costs the article event ($0.0015), because the headline, URL, publisher, date, and snippet are delivered. The run's OUTPUT record counts fullTextBlocked and fullTextPaywalled.

Sites that often return blocked from Apify's datacenter IPs include Reuters, WSJ, Bloomberg, and other wire-service, financial, and login-walled publishers. A residential proxy sometimes helps. Filter on fullTextStatus to retry only those rows.

A broken publisher never fails the run.

Input

The form is intentionally close to the popular Google News Actors: keywords, topic feeds, locale as US:en, and a time window.

Field	Default	Notes
queries	["OpenAI"]	Keywords or Google News operators (OR, site:, -, quotes, intitle:)
topics	[]	WORLD, NATION, BUSINESS, TECHNOLOGY, ENTERTAINMENT, SPORTS, SCIENCE, HEALTH
topicUrls / topicsHashed	[]	Paste a topic URL or the hash from news.google.com/topics/{id}
locale	US:en	Sets hl, gl, ceid
timeWindow	7d	1h, 1d, 7d, 30d, 1y, all → when: on keyword RSS
dateAfter / dateBefore	—	YYYY-MM-DD; replaces when: on searches; filters topic feeds
maxArticlesPerQuery	8	Google search RSS tops out around 100
deduplicate	true	Across queries, by GUID then resolved URL
extractFullText	true	Off = metadata only
createChunks	true	Heading-aware by default
chunkSize / chunkOverlap	1600 / 160	Characters
proxyConfiguration	off	Most publishers work without Apify Proxy

Example: last 24 hours of a company

{
  "queries": ["Tesla"],
  "locale": "US:en",
  "timeWindow": "1d",
  "maxArticlesPerQuery": 20,
  "extractFullText": true,
  "createChunks": true
}

Example: technology section in French

{
  "queries": [],
  "topics": ["TECHNOLOGY"],
  "locale": "FR:fr",
  "maxArticlesPerQuery": 15
}

Example: metadata only (cheap monitoring)

{
  "queries": ["\"interest rates\" OR inflation"],
  "timeWindow": "1d",
  "extractFullText": false,
  "createChunks": false
}

Example: hashed topic + date range

{
  "topicsHashed": [
    "CAAqJggKIiBDQkFTRWdvSUwyMHZNRGRqTVhZU0FtVnVHZ0pWVXlnQVAB/sections/CAQiQ0NCQVNMQW9JTDIwdk1EZGpNWFlTQW1WdUdnSlZVeUlOQ0FRYUNRb0hMMjB2TUcxcmVpb0pFZ2N2YlM4d2JXdDZLQUEqKggAKiYICiIgQ0JBU0Vnb0lMMjB2TURkak1YWVNBbVZ1R2dKVlV5Z0FQAVAB"
  ],
  "dateAfter": "2026-10-01",
  "dateBefore": "2026-10-31",
  "locale": "US:en"
}

How it works

1. Discover via public Google News RSS (/rss/search, /rss/headlines/section/topic/…, /rss/topics/{hash}). No official News API key.
2. Resolve each wrapper. Since late 2024 the article id is opaque — it is not a Base64 publisher URL. The Actor loads the RSS splash page (/rss/articles/{id}?hl&gl&ceid), reads data-n-a-sg and data-n-a-ts, and POSTs the Fbv4je / garturlreq RPC to https://news.google.com/_/DotsSplashUi/data/batchexecute. /articles/{id} and /read/{id} are not used; those 302 to google.com/sorry from non-browser clients.
3. Extract the publisher page with trafilatura and emit Markdown.
4. Chunk on headings, then on paragraphs, with configurable size and overlap. Offsets point back into the Markdown string.
5. Charge pay-per-event: run-start once, article on every row, full-text-article only when status is ok. If you set a maximum charge for the run, the Actor stops cleanly instead of overrunning.

Requests are concurrent with a polite global (and per-host) delay. One failed page does not fail the run.

Pricing

Pay-per-event (you are not billed platform compute as a user of a Store Actor):

Event	When	Price
run-start	Once at the beginning of every run	$0.002
article	Every dataset row	$0.0015
full-text-article	Only if Markdown was extracted	$0.003

A typical 100-article run with a 70% full-text hit rate costs about $0.36 ($0.002 + 100 × $0.0015 + 70 × $0.003). Paywalled rows stay at $0.0015 and still include the run-start fee. See PRICING.md for compute cost, margins, and how we measured them.

Set Max total charge on the run if you want a hard budget. The Actor bills run-start first, then checks remaining article and full-text-article events and exits without starting work it cannot bill.

Output views

The dataset ships with three Console views:

• Overview — title, publisher, URL, date, words, status
• Markdown — the article body
• Chunks — one row per chunk (unwind on chunks)

Download JSON, CSV, or Excel, or pull the dataset from the Apify API / MCP.

RAG usage

Chunks are ready for a vector store. Example mapping:

for item in dataset_items:
    for chunk in item.get("chunks") or []:
        metadata = {
            "source": chunk["source"],
            "section": chunk["section"],
            "title": item["title"],
            "publisher": item["publisher"],
            "publishedAt": item["publishedAt"],
            "chunkIndex": chunk["chunkIndex"],
            "startChar": chunk["startChar"],
            "endChar": chunk["endChar"],
        }
        # upsert chunk["text"] + metadata

Same field names as document-to-markdown-rag, so one ingest path can handle PDFs and news.

More actors from this developer

• PDF to Markdown: DOCX, PPTX, XLSX, OCR & RAG Chunks: PDF, DOCX, PPTX, XLSX and HTML to clean Markdown and RAG-ready chunks.
• Google Trends Scraper & API: Interest, Related, Trending: interest over time, by region, and related queries from Google Trends.
• Greenhouse, Lever, Ashby & Workable Jobs API: one clean jobs feed from company job boards, with salaries when posted.
• Website Screenshot & URL to PDF API (Bulk, Full Page): full-page and mobile screenshots (PNG, JPEG, WebP) and PDFs of any URL.
• Wayback Machine API: Snapshots & Archived Pages: every archived capture of a URL or domain, the snapshot closest to a date, and archived pages as Markdown.

Local development

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python -m src

Put input in storage/key_value_stores/default/INPUT.json (or run via the Apify CLI: apify run). Unit tests use fixtures and do not hit the network:

$pytest -q

Live smoke (Google News + publishers):

$RUN_LIVE_TESTS=1 pytest -q -m live

Default Store input (queries: ["NASA"], last 7 days, 8 articles, full text on) is sized to finish in well under five minutes so Apify's daily default-input check stays green. NASA.gov pages extract reliably without a proxy.

FAQ

Do I need an API key? No. The Actor uses the public Google News RSS feeds.

Why is url sometimes null? Google changed wrapper decoding in 2024. If the splash page omits data-n-a-sg / data-n-a-ts, or batchexecute refuses the RPC, the row is decode_failed. RSS metadata is still there.

Why is status metadata_only? Check fullTextStatus. blocked means the publisher refused the scraper (for example HTTP 403). paywalled, fetch_failed, and too_short cover the other cases. You are not charged the full-text event for any of them.

Can I turn off full text? Yes. Set extractFullText to false. You still get titles, publishers, dates, and decoded URLs (when resolution works).

Does it use a browser? No. RSS + HTTP + trafilatura. That keeps memory at 1 GB and the run cheap.

Should I enable Apify Proxy? Leave it off first. Enable residential or datacenter proxy only if Google News or a specific publisher starts blocking the platform IPs. Open publishers (NASA, BBC, NPR, Nature, WHO, UN, ABC News) usually work. Wire-service and login walls (Reuters, WSJ, OpenAI.com) often return metadata_only without a proxy — and you are not charged for full text on those rows.

How is this different from fetching the Google News link in a browser? A browser follows Google's redirect. Non-browser clients hitting /articles/{id} get a sorry page. The RSS splash + batchexecute path is the reliable machine method as of 2026.

What about copyright? Google's RSS notice limits the feed to personal, non-commercial use of the feed. You are responsible for how you use publisher article text. Prefer linking, quoting under applicable exceptions, or working with sources you have rights to.

Will the default input always return results? It searches NASA over the last 7 days. NASA.gov articles extract without a login wall, which keeps the daily Store smoke test green. If Google News is unreachable, the run succeeds with an empty dataset and a warning.

Publishing

This repository is the Actor source. Deploy steps, PPE event names, and the source tarball layout are in PUBLISHING.md.