# Google News Scraper: Articles, Publisher URLs & Full Text (`rjmacarthy/google-news-scraper`) Actor

Scrape Google News search results, topics and top stories in any edition. Real publisher URLs included at no extra charge, date slicing past the 100-result cap, optional full article text, and an only-new-articles mode for news monitoring.

- **URL**: https://apify.com/rjmacarthy/google-news-scraper.md
- **Developed by:** [rj](https://apify.com/rjmacarthy) (community)
- **Categories:** News, Automation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 article scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Google News Scraper: Articles, Publisher URLs & Full Text

**Google News Scraper** collects news articles from **Google News**: search results (with all the Google News operators), topic sections and top stories, in **75 country/language editions**. Every article comes with its headline, publisher, publication time and the **real publisher URL**, not the `news.google.com/rss/articles/...` redirect link that Google's feed gives you.

What makes it different from other Google News scrapers:

- 🔗 **Real publisher URLs, included in the price.** Google News links are encrypted redirects. This Actor resolves them to the actual article URL (e.g. `https://www.cnbc.com/2026/11/02/...html`) at no extra charge. Other scrapers either give you the Google redirect or charge a separate event for each resolved URL.
- 📅 **More than 100 results per query.** Google News returns at most ~100 articles per search. Set a date range and the Actor runs **one search per day** (date slicing), then merges the days and removes duplicates. In our test, one query over 3 days returned **263 unique articles** instead of 100.
- 📰 **Optional full article text.** Turn on full text to get the main text, author, published time, main image and description from the publisher page. You pay for text **only when it was actually extracted**: paywalled and blocked pages are flagged and cost nothing extra.
- 🔔 **Monitoring mode.** "Only new articles" remembers what earlier runs returned, per query and edition, so a scheduled run gives you just the new articles. It works well for brand alerts, competitor tracking and news feeds.

No login, no browser, no proxy needed by default. You pay per article.

### What data can you get?

| Field | Example | Notes |
|---|---|---|
| `title` | Defense, healthcare, crypto and AI: How the midterm election could affect stocks | Without the " - Publisher" suffix |
| `publisher`, `publisherHomepage`, `publisherDomain` | CNBC, https://www.cnbc.com, cnbc.com | From Google News |
| `publisherUrl` | https://www.cnbc.com/2026/11/02/midterm-election-stocks-defense-healthcare-crypto-ai.html | **Real article URL** (included in the price) |
| `publisherUrlResolved` | `true` | `false` if Google refused to resolve the link (`publisherUrl` is then the Google link) |
| `googleNewsUrl`, `googleNewsId` | https://news.google.com/rss/articles/CBMi... | The original Google News link |
| `publishedAt` | 2026-11-02T11:00:01.000Z | ISO 8601, UTC |
| `query` / `topic` / `sourceType` | artificial intelligence / null / search | What returned the article |
| `rank` | 3 | Position in the Google News feed |
| `relatedArticles`, `relatedArticlesCount` | \[{ title, publisher, googleNewsUrl }, ...] | Other coverage Google groups with the story (topic feeds and top stories) |
| `searchDay` | 2026-10-31 | With date slicing: the day searched |
| `edition`, `language`, `country` | US:en, en-US, US | Google News edition |
| `snippet` | null | Google News RSS no longer includes summaries; filled only when the feed has text beyond the headline |
| **Full text mode:** `text`, `wordCount` | "Lockheed Martin gets about 70% of its sales..." , 1547 | Main article text, paragraphs separated by blank lines |
| `author`, `articlePublishedAt`, `imageUrl`, `description` | Luke Fountain, 2026-11-02T11:00:01Z, https://image.cnbcfm.com/..., "Democratic gains in Congress..." | From the publisher page (meta tags and JSON-LD) |
| `textExtracted`, `textError`, `paywalled` | true, null, false | Why text is missing: `paywall`, `blocked (HTTP 403)`, ... |
| `id`, `scrapedAt` | b2ecca80b781999a2e1f | Stable ID (hash of the publisher URL) |

### Use cases

- **Media monitoring and PR:** track every mention of your brand, CEO or product, scheduled hourly, and receive only new articles.
- **Competitor and market intelligence:** follow competitors, deals and regulation with `site:`, `intitle:` and exact-phrase queries.
- **Financial and trading signals:** news flow per ticker or company with exact timestamps and publisher URLs.
- **AI and LLM pipelines:** build RAG corpora and news summaries, and train classifiers on full article text with real source URLs.
- **Research and journalism:** collect a month of coverage on a topic, day by day, across countries and languages.
- **News apps and newsletters:** current top stories and section feeds in 75 editions, with related coverage per story.

### How to use it

1. Enter one or more **search queries**, and/or pick **topic feeds** (Top stories, World, Business, Technology...).
2. Choose the **edition** (country and language), e.g. United States (English), United Kingdom, Germany, India, Brazil.
3. Set **Max articles per query**, then click **Start**.

#### Google News search operators

All of them work inside a query:

| Operator | Example | Meaning |
|---|---|---|
| `"..."` | `"electric vehicles"` | Exact phrase |
| `site:` | `tesla site:reuters.com` | Only one publisher |
| `intitle:` | `intitle:nvidia` | Word in the headline |
| `-` / `OR` | `apple -fruit`, `openai OR anthropic` | Exclude / either term |
| `when:` | `bitcoin when:7d`, `when:12h` | Last N hours/days/months |
| `after:` / `before:` | `after:2026-10-01 before:2026-10-15` | Date range (US Pacific days; `before:` is exclusive) |

#### Examples

Latest articles for a query (the prefilled input):

```json
{
    "queries": ["artificial intelligence"],
    "edition": "US:en",
    "maxArticlesPerQuery": 10,
    "decodeUrls": true,
    "extractArticleText": false
}
```

**A whole month of coverage, past the 100-result cap (date slicing):**

```json
{
    "queries": ["\"OpenAI\""],
    "dateFrom": "30 days",
    "splitByDay": true,
    "maxArticlesPerQuery": 0
}
```

The Actor runs 31 searches (one per day, newest first), merges them and removes duplicates. A query with `when:30d` or `after:`/`before:` in it is sliced the same way.

**Top stories and sections in the UK edition, with full text:**

```json
{
    "topics": ["TOP_STORIES", "BUSINESS", "TECHNOLOGY"],
    "edition": "GB:en",
    "maxArticlesPerQuery": 30,
    "extractArticleText": true
}
```

**Hourly brand monitor (only new articles):**

```json
{
    "queries": ["\"Acme Corp\"", "\"Acme\" intitle:recall"],
    "edition": "US:en",
    "onlyNewArticles": true,
    "stateKey": "acme-monitor"
}
```

Schedule it in **Schedules** (e.g. every hour) and connect a webhook, Slack, email or Zapier to the run's dataset. The first run returns the current articles and saves them as the baseline. Each later run returns only articles you haven't seen yet, and you pay only for those.

### Output example

Real output from a local run (`samples/sample-output.json`, text shortened):

```json
{
    "id": "b2ecca80b781999a2e1f",
    "title": "Defense, healthcare, crypto and AI: How the midterm election could affect stocks",
    "publisher": "CNBC",
    "publisherHomepage": "https://www.cnbc.com",
    "publisherDomain": "cnbc.com",
    "publisherUrl": "https://www.cnbc.com/2026/11/02/midterm-election-stocks-defense-healthcare-crypto-ai.html",
    "publisherUrlResolved": true,
    "publisherUrlError": null,
    "url": "https://www.cnbc.com/2026/11/02/midterm-election-stocks-defense-healthcare-crypto-ai.html",
    "googleNewsUrl": "https://news.google.com/rss/articles/CBMikwFBVV95cUxNNTJxMVlLZVJ1YmQwOGw0Y2R0b3JqQS04aWZz...?oc=5",
    "googleNewsId": "CBMikwFBVV95cUxNNTJxMVlLZVJ1YmQwOGw0Y2R0b3JqQS04aWZz...",
    "publishedAt": "2026-11-02T11:00:01.000Z",
    "snippet": null,
    "relatedArticles": [],
    "relatedArticlesCount": 0,
    "sourceType": "search",
    "query": "artificial intelligence",
    "topic": null,
    "searchDay": null,
    "rank": 3,
    "edition": "US:en",
    "language": "en-US",
    "country": "US",
    "textExtracted": true,
    "textError": null,
    "text": "Lockheed Martin gets about 70% of its sales from the U.S. government. Medicaid accounts for three-quarters of Molina Healthcare's premium revenue ...",
    "wordCount": 1547,
    "author": "Luke Fountain",
    "description": "Democratic gains in Congress could bring tougher oversight of major companies and shift billions of dollars in federal spending and regulatory policy.",
    "imageUrl": "https://image.cnbcfm.com/api/v1/image/106949322-1632916362498-gettyimages-1235565564-848033_NA-POL-0928-CAPITOL-HILL_KKN_29317JPG.jpg?v=1793390260&w=1920&h=1080",
    "articlePublishedAt": "2026-11-02T11:00:01.000Z",
    "articleLanguage": "en",
    "paywalled": false,
    "scrapedAt": "2026-11-02T13:28:51.246Z"
}
```

A **RUN_SUMMARY** record in the run's key-value store lists every query and feed with its status, counts, duplicates, URL-decoding and full-text statistics, and request counts.

### Pricing

Pay per event, no start fee:

| Event | Price | When |
|---|---|---|
| `article` | **$0.001** | Every article in the output, **including its resolved publisher URL** |
| `article-text` | **$0.002** | Only when full text was requested **and** successfully extracted |

Worked examples:

- **1,000 articles with publisher URLs ≈ $1.00.**
- 1,000 articles with full text, if 650 texts are extracted ≈ $1.00 + $1.30 = **$2.30**. Paywalled or blocked pages cost $0.001 (the article only).
- An hourly monitor finding 20 new articles per run ≈ $0.02 per run, about $14 a month.

Set **Maximum cost per run** on the run to cap your spend: the Actor stops cleanly when the budget is reached.

### Tips and limits

- **~100 articles per search request** is a Google limit. Use a date range with date slicing to go beyond it. If a single day still returns ~100 results (the run log says so), narrow the query (e.g. add `site:` or exact phrases) to get complete coverage of that day.
- **Date slicing costs one Google request per day** searched, and stops as soon as "Max articles per query" is reached, so raise that limit (or set 0) for full ranges. Days are US Pacific calendar days, as Google defines them.
- **Topic feeds and top stories** return Google's current selection (~40-70 articles per section). The date range applies to search queries only.
- **Resolving publisher URLs** takes about one request to Google per article (plus one per 10 articles). The Actor runs 2 in parallel and backs off automatically if Google rate-limits. If Google refuses, the article still comes out, with the Google link and `publisherUrlResolved: false`. A run never fails because of URL decoding. Resolved URLs are remembered (`google-news-url-cache` store), so repeat runs resolve known articles instantly. Turn decoding off for the fastest possible run.
- **Full text is best-effort.** It works on most open news sites. Hard paywalls (in our tests the Washington Post; the New York Times answered HTTP 403), registration walls and bot-protected sites return `text: null` with a `textError` reason, and you are not charged for their text. Metered sites that serve the full article are extracted, with `paywalled: true`. Each site gets at most one request per second.
- **Snippets:** Google News RSS no longer includes article summaries (the description repeats the headline), so `snippet` is usually null. Turn on full text to get the publisher's description and the article text.
- **Related coverage** links are Google News links (not resolved, to keep runs fast and cheap).
- **Memory:** 256 MB is enough for most runs. Use 512 MB for large full-text runs.
- **Proxy:** not needed normally. If a run reports HTTP 429/403 from Google, enable Apify Proxy (a US location is best: EU IPs can get Google's cookie-consent page).

### FAQ

**Is it legal to scrape Google News?**
This Actor reads Google News' public RSS feeds and public article pages, without logging in. It does not collect personal data beyond what publishers print publicly (such as an article's author byline). You are responsible for how you use the data: respect copyright (store and republish article text only where you have the right to), the publishers' terms, and laws such as GDPR. If in doubt, ask a lawyer.

**Why do other scrapers give me `news.google.com` links?**
Since 2024 Google News encrypts article links. Resolving them needs extra requests to Google, so many scrapers skip it or charge extra. Here it is included in the article price.

**How many articles can I get for one query?**
Without a date range: up to ~100 (Google's limit). With a date range and date slicing: up to ~100 **per day**, e.g. thousands for a month.

**How does "only new articles" work?**
The Actor stores the IDs of the articles it has output in a named key-value store (`google-news-scraper-state`), separately for each query/topic and edition, under your **State key**. Later runs with the same state key skip those articles. Use a different state key for each independent monitor.

**Can I get news in other languages and countries?**
Yes, pick one of 75 editions (e.g. Germany (Deutsch), France, Japan, Brazil, India (हिन्दी), Mexico). Run the Actor once per edition if you need several.

**Why is `publisherUrlResolved` false for some articles?**
Google sometimes refuses or rate-limits the decoding request. The article is still output with its Google News link. `publisherUrlError` tells you why.

**Does it use a browser?**
No. It uses fast HTTP requests only, so it is cheap and quick.

### Integrations

- **API:** run the Actor and fetch the dataset via the [Apify API](https://docs.apify.com/api/v2) or the JavaScript/Python clients.
- **Make, Zapier, n8n:** use the Apify app/node to start runs and read the dataset, e.g. post every new article to Slack, Google Sheets or a CRM.
- **Webhooks:** trigger your own endpoint when a run finishes.
- **MCP / AI agents:** use the Actor as a tool through the [Apify MCP server](https://mcp.apify.com) so assistants such as Claude can search Google News and read articles.
- Export to JSON, CSV, Excel, XML or RSS from the dataset tab.

### Changelog

- **0.1 (2026-11):** First release: Google News search (all operators), topic feeds and top stories in 75 editions; publisher URL decoding included (batched, cached, with graceful fallback); day-by-day date slicing past the 100-result cap; optional full text (Readability) with paywall/block detection; incremental "only new articles" mode per query and edition.

### More scrapers by the same developer

- [Substack Scraper](https://apify.com/rjmacarthy/substack-scraper)
- [Google Ads Transparency Scraper](https://apify.com/rjmacarthy/google-ads-transparency-scraper)
- [Telegram Channel Scraper](https://apify.com/rjmacarthy/telegram-channel-scraper)
- [ATS Jobs Scraper: Greenhouse, Lever, Ashby & Workday](https://apify.com/rjmacarthy/ats-jobs-scraper)
- [Dice Jobs Scraper](https://apify.com/rjmacarthy/dice-jobs-scraper)
- [SEEK Jobs Scraper](https://apify.com/rjmacarthy/seek-jobs-scraper)

Browse all my Actors on [my Apify Store profile](https://apify.com/rjmacarthy).

# Actor input Schema

## `queries` (type: `array`):

One Google News search per line. Google News operators work: "exact phrase", site:reuters.com, intitle:apple, -exclude, OR, when:7d (last 7 days), when:12h, after:2026-10-01 before:2026-10-15. Each query returns up to 100 articles per Google request; use a date range with date slicing to get more.

## `topics` (type: `array`):

Google News sections to scrape in the chosen edition: Top stories (the front page), World, Nation (your edition country), Business, Technology, Entertainment, Sports, Science, Health. Each topic feed returns its current ~40-70 articles, including the related coverage Google groups with each story.

## `edition` (type: `string`):

Which Google News edition to read: it sets the country and language of the results (Google's hl, gl and ceid parameters).

## `maxArticlesPerQuery` (type: `integer`):

Stop each search query or topic feed after this many articles. 0 = no limit. Without date slicing Google returns at most ~100 per query; with a date range and date slicing you can get thousands.

## `decodeUrls` (type: `boolean`):

Google News RSS links point to news.google.com redirect pages. When on, each article is resolved to the real publisher URL (e.g. https://www.reuters.com/...), included in the article price. Turn off for a faster run that only needs titles, publishers and dates. If Google refuses to resolve a link, the article keeps its Google News link and publisherUrlResolved is false; the run never fails because of it. Always on when full text is requested (text is fetched from the publisher URL).

## `extractArticleText` (type: `boolean`):

Open each publisher page and extract the main article text, author, published time, main image (og:image) and description. Charged as an extra "article-text" event only when the text was actually extracted: paywalled, blocked or JavaScript-only pages get text: null with a textError reason and are not charged for text. Slower: publisher pages are fetched politely (at most 1 request per second per site).

## `dateFrom` (type: `string`):

Only articles published on or after this day (US Pacific time, like Google News), e.g. 2026-10-01, or a relative value like "30 days". Applies to search queries; topic feeds always show current articles. Overrides when:/after:/before: in the queries.

## `dateTo` (type: `string`):

Only articles published on or before this day, e.g. 2026-10-31. Leave empty for today.

## `splitByDay` (type: `boolean`):

When a date range is set (here or with when:7d / after: / before: in a query), split it into one search per day, newest day first, and merge the results. Each day can return up to ~100 articles. Costs one Google request per day searched; it stops as soon as "Max articles per query" is reached.

## `onlyNewArticles` (type: `boolean`):

Output (and charge) only articles that previous runs with the same state key did not output, tracked per query/topic and edition. The first run outputs everything and becomes the baseline. Schedule the Actor (e.g. every hour) for news alerts.

## `stateKey` (type: `string`):

Name of this monitor. Runs with the same state key share their memory of seen articles; use a different key for each independent monitor.

## `deduplicateAcrossQueries` (type: `boolean`):

Output each article only once per run, even if several queries or topics return it (it is reported under the first one). Turn off to get the article once per matching query.

## `cacheDecodedUrls` (type: `boolean`):

Keep resolved publisher URLs in a named key-value store (google-news-url-cache) so later runs that see the same articles resolve them instantly, with fewer requests to Google.

## `maxConcurrency` (type: `integer`):

How many requests to news.google.com run in parallel. Keep it low (2) to avoid rate limiting; the Actor also slows down automatically when Google answers HTTP 429.

## `textConcurrency` (type: `integer`):

How many publisher pages are fetched in parallel for full text (never more than one at a time, and one per second, for the same site).

## `proxyConfiguration` (type: `object`):

Not needed normally. If Google starts answering HTTP 429 / 403 for your runs, enable Apify Proxy (a US proxy is best; EU IPs may get Google's cookie-consent page).

## Actor input object example

```json
{
  "queries": [
    "artificial intelligence",
    "\"OpenAI\" site:reuters.com",
    "tesla when:7d"
  ],
  "edition": "US:en",
  "maxArticlesPerQuery": 10,
  "decodeUrls": true,
  "extractArticleText": false,
  "splitByDay": true,
  "onlyNewArticles": false,
  "stateKey": "default",
  "deduplicateAcrossQueries": true,
  "cacheDecodedUrls": true,
  "maxConcurrency": 2,
  "textConcurrency": 4,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `articles` (type: `string`):

One item per article: headline, publisher, published time and the real publisher URL.

## `articlesFull` (type: `string`):

Every field, including related coverage, edition details and full text when enabled.

## `summary` (type: `string`):

Per-feed status and counts, URL decoding and full-text statistics, request counts.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "artificial intelligence"
    ],
    "edition": "US:en",
    "maxArticlesPerQuery": 10,
    "decodeUrls": true,
    "extractArticleText": false
};

// Run the Actor and wait for it to finish
const run = await client.actor("rjmacarthy/google-news-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "queries": ["artificial intelligence"],
    "edition": "US:en",
    "maxArticlesPerQuery": 10,
    "decodeUrls": True,
    "extractArticleText": False,
}

# Run the Actor and wait for it to finish
run = client.actor("rjmacarthy/google-news-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "artificial intelligence"
  ],
  "edition": "US:en",
  "maxArticlesPerQuery": 10,
  "decodeUrls": true,
  "extractArticleText": false
}' |
apify call rjmacarthy/google-news-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,rjmacarthy/google-news-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/HbP67071cFz1qh0ok/builds/gQGGVm1AtVFELz6Q4/openapi.json