# Google News Scraper: Full-Text Articles to Markdown (`sunburst077/google-news-to-markdown`) Actor

Find Google News articles by keyword or topic and return each full article as clean Markdown plus RAG-ready chunks.

- **URL**: https://apify.com/sunburst077/google-news-to-markdown.md
- **Developed by:** [Sunburst](https://apify.com/sunburst077) (community)
- **Categories:** News, AI, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 articles

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Google News Scraper: Full-Text Articles to Markdown

Search [Google News](https://news.google.com) by keyword or topic and get every article's full text as clean Markdown, plus RAG-ready chunks.

The Actor resolves each `news.google.com/rss/articles/…` wrapper to the publisher URL, fetches the page, and extracts the main article with [trafilatura](https://github.com/adbar/trafilatura). You get structured metadata plus Markdown — not just headlines and links.

It is built for AI/RAG pipelines and pairs with the same developer's [document-to-markdown-rag](https://apify.com/sunburst077/document-to-markdown-rag) Actor. Chunks use the same shape: `text`, `source`, `section`, `chunkIndex`, `startChar`, `endChar`.

### Why this Actor

Headline scrapers such as [data_xplorer/google-news-scraper-fast](https://apify.com/data_xplorer/google-news-scraper-fast) and [lhotanova/google-news-scraper](https://apify.com/lhotanova/google-news-scraper) are excellent for monitoring titles and dates. They stop at the Google News card (and sometimes a decoded URL or image).

This Actor goes one step further:

| | Headline scrapers | **This Actor** |
| --- | --- | --- |
| Title, publisher, date | Yes | Yes |
| Resolved publisher URL | Often an add-on | Always (when Google still serves the splash) |
| Full article body | No | **Markdown** |
| RAG chunks | No | **Heading-aware**, with offsets |
| Paywalled / blocked pages | Usually a failed row | Metadata kept, **full-text event not charged** |
| Price | about $0.004 / headline | **$0.002 / run** + **$0.0015 / article** + **$0.003** only when full text succeeds |

Use a headline scraper when you only need the feed. Use this Actor when you want to embed, summarize, or cite the article itself.

### What you get

Each dataset item:

```json
{
  "title": "Introducing GPT-6.1 Terra",
  "publisher": "OpenAI",
  "publisherUrl": "https://openai.com",
  "url": "https://openai.com/index/introducing-gpt-6-1-terra/",
  "googleLink": "https://news.google.com/rss/articles/CBMi…",
  "publishedAt": "2026-10-26T18:05:36Z",
  "author": null,
  "language": "en",
  "description": "Introducing GPT-6.1 Terra  OpenAI",
  "markdown": "# Introducing GPT-6.1 Terra\n\n…",
  "wordCount": 842,
  "chunks": [
    {
      "text": "# Introducing GPT-6.1 Terra\n\n…",
      "source": "https://openai.com/index/introducing-gpt-6-1-terra/",
      "section": "Introducing GPT-6.1 Terra",
      "chunkIndex": 0,
      "startChar": 0,
      "endChar": 1580
    }
  ],
  "status": "ok",
  "statusReason": null,
  "fullTextStatus": "extracted",
  "httpStatus": 200,
  "query": "OpenAI",
  "sourceType": "keyword"
}
```

`status` is one of:

- `ok` — full Markdown extracted; the `full-text-article` event is charged
- `metadata_only` — paywall, bot challenge, short/empty body, or full text turned off; only the `article` event is charged
- `decode_failed` — Google News wrapper could not be resolved; metadata from RSS is still returned

`fullTextStatus` says what happened when the Actor fetched the full text:

| `fullTextStatus` | Meaning | `full-text-article` charged? |
| --- | --- | --- |
| `extracted` | Markdown extracted (`status: ok`) | **Yes** |
| `blocked` | The publisher refused the scraper: HTTP 401, 402, 403, 407, 429, or 451, or a bot-challenge page ("checking your browser", "access denied") | No |
| `paywalled` | Subscriber wall | No |
| `fetch_failed` | Network error or another 4xx/5xx (`httpStatus` has the code) | No |
| `too_short` | Page loaded but had less usable text than `minFullTextChars` | No |
| `failed` | The extractor crashed on this page | No |
| `skipped_budget` | Your max charge for the run was reached, so the body was withheld | No |
| `disabled` | `extractFullText` is off | No |
| `not_attempted` | The Google News link could not be decoded | No |

`httpStatus` is the HTTP status code the publisher returned. Only a row with `fullTextStatus: "extracted"` and non-empty `markdown` is billed for full text. The charging code checks this again just before it bills. A blocked row still costs the `article` event ($0.0015), because the headline, URL, publisher, date, and snippet are delivered. The run's `OUTPUT` record counts `fullTextBlocked` and `fullTextPaywalled`.

Sites that often return `blocked` from Apify's datacenter IPs include Reuters, WSJ, Bloomberg, and other wire-service, financial, and login-walled publishers. A residential proxy sometimes helps. Filter on `fullTextStatus` to retry only those rows.

A broken publisher never fails the run.

### Input

The form is intentionally close to the popular Google News Actors: keywords, topic feeds, locale as `US:en`, and a time window.

| Field | Default | Notes |
| --- | --- | --- |
| `queries` | `["OpenAI"]` | Keywords or Google News operators (`OR`, `site:`, `-`, quotes, `intitle:`) |
| `topics` | `[]` | `WORLD`, `NATION`, `BUSINESS`, `TECHNOLOGY`, `ENTERTAINMENT`, `SPORTS`, `SCIENCE`, `HEALTH` |
| `topicUrls` / `topicsHashed` | `[]` | Paste a topic URL or the hash from `news.google.com/topics/{id}` |
| `locale` | `US:en` | Sets `hl`, `gl`, `ceid` |
| `timeWindow` | `7d` | `1h`, `1d`, `7d`, `30d`, `1y`, `all` → `when:` on keyword RSS |
| `dateAfter` / `dateBefore` | — | `YYYY-MM-DD`; replaces `when:` on searches; filters topic feeds |
| `maxArticlesPerQuery` | `8` | Google search RSS tops out around 100 |
| `deduplicate` | `true` | Across queries, by GUID then resolved URL |
| `extractFullText` | `true` | Off = metadata only |
| `createChunks` | `true` | Heading-aware by default |
| `chunkSize` / `chunkOverlap` | `1600` / `160` | Characters |
| `proxyConfiguration` | off | Most publishers work without Apify Proxy |

#### Example: last 24 hours of a company

```json
{
  "queries": ["Tesla"],
  "locale": "US:en",
  "timeWindow": "1d",
  "maxArticlesPerQuery": 20,
  "extractFullText": true,
  "createChunks": true
}
```

#### Example: technology section in French

```json
{
  "queries": [],
  "topics": ["TECHNOLOGY"],
  "locale": "FR:fr",
  "maxArticlesPerQuery": 15
}
```

#### Example: metadata only (cheap monitoring)

```json
{
  "queries": ["\"interest rates\" OR inflation"],
  "timeWindow": "1d",
  "extractFullText": false,
  "createChunks": false
}
```

#### Example: hashed topic + date range

```json
{
  "topicsHashed": [
    "CAAqJggKIiBDQkFTRWdvSUwyMHZNRGRqTVhZU0FtVnVHZ0pWVXlnQVAB/sections/CAQiQ0NCQVNMQW9JTDIwdk1EZGpNWFlTQW1WdUdnSlZVeUlOQ0FRYUNRb0hMMjB2TUcxcmVpb0pFZ2N2YlM4d2JXdDZLQUEqKggAKiYICiIgQ0JBU0Vnb0lMMjB2TURkak1YWVNBbVZ1R2dKVlV5Z0FQAVAB"
  ],
  "dateAfter": "2026-10-01",
  "dateBefore": "2026-10-31",
  "locale": "US:en"
}
```

### How it works

1. **Discover** via public Google News RSS (`/rss/search`, `/rss/headlines/section/topic/…`, `/rss/topics/{hash}`). No official News API key.
2. **Resolve** each wrapper. Since late 2024 the article id is opaque — it is **not** a Base64 publisher URL. The Actor loads the RSS splash page (`/rss/articles/{id}?hl&gl&ceid`), reads `data-n-a-sg` and `data-n-a-ts`, and POSTs the `Fbv4je` / `garturlreq` RPC to `https://news.google.com/_/DotsSplashUi/data/batchexecute`. `/articles/{id}` and `/read/{id}` are not used; those 302 to `google.com/sorry` from non-browser clients.
3. **Extract** the publisher page with trafilatura and emit Markdown.
4. **Chunk** on headings, then on paragraphs, with configurable size and overlap. Offsets point back into the Markdown string.
5. **Charge** pay-per-event: `run-start` once, `article` on every row, `full-text-article` only when `status` is `ok`. If you set a maximum charge for the run, the Actor stops cleanly instead of overrunning.

Requests are concurrent with a polite global (and per-host) delay. One failed page does not fail the run.

### Pricing

Pay-per-event (you are not billed platform compute as a user of a Store Actor):

| Event | When | Price |
| --- | --- | --- |
| `run-start` | Once at the beginning of every run | **$0.002** |
| `article` | Every dataset row | **$0.0015** |
| `full-text-article` | Only if Markdown was extracted | **$0.003** |

A typical 100-article run with a 70% full-text hit rate costs about **$0.36** (`$0.002 + 100 × $0.0015 + 70 × $0.003`). Paywalled rows stay at $0.0015 and still include the run-start fee. See [PRICING.md](PRICING.md) for compute cost, margins, and how we measured them.

Set **Max total charge** on the run if you want a hard budget. The Actor bills `run-start` first, then checks remaining `article` and `full-text-article` events and exits without starting work it cannot bill.

### Output views

The dataset ships with three Console views:

- **Overview** — title, publisher, URL, date, words, status
- **Markdown** — the article body
- **Chunks** — one row per chunk (`unwind` on `chunks`)

Download JSON, CSV, or Excel, or pull the dataset from the Apify API / MCP.

### RAG usage

Chunks are ready for a vector store. Example mapping:

```python
for item in dataset_items:
    for chunk in item.get("chunks") or []:
        metadata = {
            "source": chunk["source"],
            "section": chunk["section"],
            "title": item["title"],
            "publisher": item["publisher"],
            "publishedAt": item["publishedAt"],
            "chunkIndex": chunk["chunkIndex"],
            "startChar": chunk["startChar"],
            "endChar": chunk["endChar"],
        }
        # upsert chunk["text"] + metadata
```

Same field names as document-to-markdown-rag, so one ingest path can handle PDFs and news.

### More actors from this developer

- [PDF to Markdown: DOCX, PPTX, XLSX, OCR & RAG Chunks](https://apify.com/sunburst077/document-to-markdown-rag): PDF, DOCX, PPTX, XLSX and HTML to clean Markdown and RAG-ready chunks.
- [Google Trends Scraper & API: Interest, Related, Trending](https://apify.com/sunburst077/google-trends-reliable): interest over time, by region, and related queries from Google Trends.
- [Greenhouse, Lever, Ashby & Workable Jobs API](https://apify.com/sunburst077/multi-ats-jobs-feed): one clean jobs feed from company job boards, with salaries when posted.
- [Website Screenshot & URL to PDF API (Bulk, Full Page)](https://apify.com/sunburst077/website-screenshot-pdf): full-page and mobile screenshots (PNG, JPEG, WebP) and PDFs of any URL.
- [Wayback Machine API: Snapshots & Archived Pages](https://apify.com/sunburst077/wayback-machine-snapshots): every archived capture of a URL or domain, the snapshot closest to a date, and archived pages as Markdown.

### Local development

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python -m src
```

Put input in `storage/key_value_stores/default/INPUT.json` (or run via the Apify CLI: `apify run`). Unit tests use fixtures and do not hit the network:

```bash
pytest -q
```

Live smoke (Google News + publishers):

```bash
RUN_LIVE_TESTS=1 pytest -q -m live
```

Default Store input (`queries: ["NASA"]`, last 7 days, 8 articles, full text on) is sized to finish in well under five minutes so Apify's daily default-input check stays green. NASA.gov pages extract reliably without a proxy.

### FAQ

**Do I need an API key?**
No. The Actor uses the public Google News RSS feeds.

**Why is `url` sometimes null?**
Google changed wrapper decoding in 2024. If the splash page omits `data-n-a-sg` / `data-n-a-ts`, or batchexecute refuses the RPC, the row is `decode_failed`. RSS metadata is still there.

**Why is `status` `metadata_only`?**
Check `fullTextStatus`. `blocked` means the publisher refused the scraper (for example HTTP 403). `paywalled`, `fetch_failed`, and `too_short` cover the other cases. You are not charged the full-text event for any of them.

**Can I turn off full text?**
Yes. Set `extractFullText` to `false`. You still get titles, publishers, dates, and decoded URLs (when resolution works).

**Does it use a browser?**
No. RSS + HTTP + trafilatura. That keeps memory at 1 GB and the run cheap.

**Should I enable Apify Proxy?**
Leave it off first. Enable residential or datacenter proxy only if Google News or a specific publisher starts blocking the platform IPs. Open publishers (NASA, BBC, NPR, Nature, WHO, UN, ABC News) usually work. Wire-service and login walls (Reuters, WSJ, OpenAI.com) often return `metadata_only` without a proxy — and you are not charged for full text on those rows.

**How is this different from fetching the Google News link in a browser?**
A browser follows Google's redirect. Non-browser clients hitting `/articles/{id}` get a sorry page. The RSS splash + batchexecute path is the reliable machine method as of 2026.

**What about copyright?**
Google's RSS notice limits the feed to personal, non-commercial use of the *feed*. You are responsible for how you use publisher article text. Prefer linking, quoting under applicable exceptions, or working with sources you have rights to.

**Will the default input always return results?**
It searches `NASA` over the last 7 days. NASA.gov articles extract without a login wall, which keeps the daily Store smoke test green. If Google News is unreachable, the run succeeds with an empty dataset and a warning.

### Publishing

This repository is the Actor source. Deploy steps, PPE event names, and the source tarball layout are in [PUBLISHING.md](PUBLISHING.md).

# Actor input Schema

## `queries` (type: `array`):

Search Google News the same way you would in the search box. Operators such as OR, quotes, minus, site:, and intitle: work. Each query is fetched as its own RSS search.

## `topics` (type: `array`):

Built-in Google News sections. These are standalone feeds, not combined with your keywords. Date filters are applied after download because topic RSS cannot take when:.

## `topicUrls` (type: `array`):

Paste a Google News topic or section URL from your browser (news.google.com/topics/... or .../sections/...). Converted to the public RSS feed automatically.

## `topicsHashed` (type: `array`):

Topic hashes from a Google News URL, e.g. CAAqJggKIiBDQkFTRWdvSUwyMHZNRGRqTVhZU0FtVnVHZ0pWVXlnQVAB or TOPIC/sections/SECTION.

## `locale` (type: `string`):

Google News edition as country:language (ceid). Sets hl, gl, and ceid together.

## `language` (type: `string`):

Optional Google News hl (e.g. en-US). Leave empty to derive it from the locale.

## `country` (type: `string`):

Optional Google News gl / country code (e.g. US). Leave empty to derive it from the locale.

## `timeWindow` (type: `string`):

Added to keyword searches as when:1d / when:7d / …. Ignored when After / Before dates are set. Topic feeds are filtered client-side.

## `dateAfter` (type: `string`):

YYYY-MM-DD. Added as after: on keyword searches. Also filters topic results.

## `dateBefore` (type: `string`):

YYYY-MM-DD. Added as before: on keyword searches. Also filters topic results.

## `maxArticlesPerQuery` (type: `integer`):

Cap per keyword or topic feed. Google News search RSS returns at most ~100 items.

## `maxArticles` (type: `integer`):

Optional global cap after de-duplication. Leave empty to keep every per-query result.

## `deduplicate` (type: `boolean`):

Drop the same Google News item (and the same resolved publisher URL) when it appears in more than one query.

## `extractFullText` (type: `boolean`):

Fetch the publisher page and convert the main article to Markdown. Turn off for metadata-only runs (cheaper, faster).

## `createChunks` (type: `boolean`):

Split Markdown into heading-aware chunks: text, source, section, chunkIndex, startChar, endChar.

## `headingAware` (type: `boolean`):

Split on Markdown headings and keep the heading path in section. Disable for a plain sliding window.

## `chunkSize` (type: `integer`):

Target maximum characters per chunk. About 4 characters per token.

## `chunkOverlap` (type: `integer`):

Characters repeated from the previous chunk when a section is split.

## `minFullTextChars` (type: `integer`):

Shorter extractions are treated as paywalled or empty (status metadata_only, no full-text charge).

## `concurrency` (type: `integer`):

Parallel HTTP requests. Keep this modest to stay polite to publishers.

## `requestDelayMs` (type: `integer`):

Global polite pause. Hosts get a slightly longer gap.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for each HTTP request before giving up.

## `maxRetries` (type: `integer`):

Retries for timeouts and 429 / 5xx responses.

## `proxyConfiguration` (type: `object`):

Optional. Off by default — most publisher pages and Google News RSS work without a proxy. Turn on Apify Proxy only if you see blocks.

## Actor input object example

```json
{
  "queries": [
    "NASA"
  ],
  "topics": [],
  "topicUrls": [],
  "topicsHashed": [],
  "locale": "US:en",
  "timeWindow": "7d",
  "maxArticlesPerQuery": 8,
  "deduplicate": true,
  "extractFullText": true,
  "createChunks": true,
  "headingAware": true,
  "chunkSize": 1600,
  "chunkOverlap": 160,
  "minFullTextChars": 240,
  "concurrency": 4,
  "requestDelayMs": 250,
  "requestTimeoutSecs": 25,
  "maxRetries": 3,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `articles` (type: `string`):

Dataset of article rows: title, publisher, resolved URL, Markdown, chunks, and extraction status.

## `summary` (type: `string`):

Counts of articles found, decoded, extracted, and charged events.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "NASA"
    ],
    "locale": "US:en",
    "timeWindow": "7d",
    "maxArticlesPerQuery": 8,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("sunburst077/google-news-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "queries": ["NASA"],
    "locale": "US:en",
    "timeWindow": "7d",
    "maxArticlesPerQuery": 8,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("sunburst077/google-news-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "NASA"
  ],
  "locale": "US:en",
  "timeWindow": "7d",
  "maxArticlesPerQuery": 8,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call sunburst077/google-news-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,sunburst077/google-news-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/aETijWoyhoSqI4Gtg/builds/m9sc57W89zKc5YAPa/openapi.json