# Google News Scraper (`scrapyx/google-news-scraper`) Actor

Scrapes Google News across 47 country/language editions: top stories, keyword search, the 8 topic sections, local news by place, topic and publisher IDs. Resolves Google's opaque redirect links into real publisher article URLs. HTTP only, no API key.

- **URL**: https://apify.com/scrapyx/google-news-scraper.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** News
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.10 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Google News Scraper

Pulls Google News across **47 country/language editions** — top stories, keyword
search, the eight topic sections, and local news by place — and turns Google's
opaque redirect links into **real publisher article URLs**.

| | |
| --- | --- |
| **Surfaces** | Top stories · keyword search · 8 topics · local news by place · raw topic IDs · publisher IDs |
| **Editions** | 47 country/language editions, from `US:en` to `JP:ja`, `ID:id`, `BR:pt-419`, `EG:ar` |
| **Returns** | headline, publisher, published time, edition — plus the resolved publisher URL and domain |
| **Method** | HTTP only. No API key, no login, no browser |
| **Anti-bot** | None on this surface — every feed answers 200 cold over plain TLS |

### The bit that actually matters: real article URLs

Every link in a Google News feed is an opaque token like
`CBMiswFBVV95cUxQcDN2djctWGl2...`. It is **not** the publisher's URL, and it
cannot be decoded offline — base64-decoding it just yields a second opaque
token. The pre-2024 trick that unpacked a plain URL is dead.

This actor resolves it properly, HTTP-only, and gives you:

```json
"articleUrl":    "https://www.politico.com/news/2026/09/08/supreme-court-rejects-...",
"articleDomain": "www.politico.com",
"googleNewsUrl": "https://news.google.com/rss/articles/CBMinwFBVV95cUxOUlZobkox..."
```

**It is not free.** Google embeds the per-article signature at the very end of a
\~600 KB page, ignores `Range` requests, and puts no signatures on any listing
page — so resolution costs one ~600 KB fetch **per article** (gzipped on the
wire, fetched concurrently). Set `resolveArticleUrls: false` for fast
headline-only sweeps; you keep the publisher's name and homepage, just not the
article link.

### Example input

```json
{
  "includeTopStories": true,
  "queries": ["artificial intelligence", "\"supply chain\" when:7d"],
  "topics": ["TECHNOLOGY", "BUSINESS"],
  "locations": ["Jakarta"],
  "editions": ["US:en", "GB:en", "ID:id"],
  "resolveArticleUrls": true,
  "maxItemsPerFeed": 50
}
```

Every selection is fetched **once per edition**, so the run above is
5 feed specs × 3 editions = 15 feeds.

### Search operators that work

Verified against the live service — these genuinely filter:

| Operator | Example | Note |
| --- | --- | --- |
| exact phrase | `"tesla model y"` | |
| `OR` | `tesla OR rivian` | |
| exclude | `tesla -musk` | |
| `site:` | `tesla site:reuters.com` | collapses to a single publisher |
| `intitle:` | `intitle:tesla` | |
| `when:` | `tesla when:1h`, `when:7d` | the cheapest way to slice a big topic |
| `after:` / `before:` | `tesla after:2026-09-01 before:2026-09-05` | |

**`allinurl:` is not supported** — it returns zero results rather than an error.

### Output

Every row carries the portfolio envelope — `_input`, `_source`, `_scrapedAt`,
`recordType`. Three record types share the dataset:

- **`ARTICLE`** — one per item: headline (with Google's ` - Publisher` suffix
  stripped, raw kept), publisher, published time, edition, and the resolved
  `articleUrl` when resolution ran.
- **`FEED_SUMMARY`** — one per feed × edition: how many items the feed held,
  how many were emitted, how many URLs resolved or failed, plus
  `requestsUsed` and `bytesDownloaded` so the cost of a run is visible in the
  data rather than guessed.
- **`ERROR`** — one per feed that failed, so **every input maps to at least one
  row**.

### Known limits — read these, they are structural

- **There is no pagination. At all.** `num`, `count`, `start`, `page`,
  `max-results` and `n` are every one of them ignored — all return the identical
  body. A feed gives what it gives (~38 top stories, ~100 search, 50–70 topic,
  20–90 local) and that is the ceiling for one request. **Breadth comes from
  more feeds, not deeper paging**: add editions, split a query with `when:`/
  `after:`, or search narrower terms. `maxItemsPerFeed` can only *reduce*
  results.
- **No article body.** Google News is an index of other people's journalism; the
  text lives on the publisher's site, behind their own paywalls and bot rules.
  This actor gives you the link, not the article.
- **No Danish edition.** Google does not have one. Every Danish URL spelling
  returns the *Norwegian* feed with `<language>no</language>`, so `DK:da` is
  deliberately excluded rather than shipped as a lookalike.
- **Two upstream failures look like success**, and are caught: an unrecognised
  topic returns 200 with 1.68 MB of HTML, and an unrecognised place returns 200
  with a one-item "This feed is not available." stub. Both become `ERROR` rows.
- **A search with no matches is not an error** — it is `returnedCount: 0` with a
  summary row.

### Licensing — please read before publishing anything built on this

Google's `robots.txt` puts `ClaudeBot`, `anthropic-ai` and `Claude-Web` under a
bare `Disallow: /`, and the feed carries its own terms in a `<copyright>`
element: made available *"solely for the purpose of rendering Google News
results within a personal feed reader for personal, non-commercial use. Any
other use of the feed is expressly prohibited."*

Unlike a publisher API there is no commercial tier to buy — Google retired the
Google News API in 2011. Those terms are Google's position on this data, they
are not enforced technically, and anyone running this actor should decide for
themselves whether their use fits. It is stated here plainly rather than left
for you to find later.

# Actor input Schema

## `includeTopStories` (type: `boolean`):

Fetch the Top stories front page for every selected edition. This is the fastest way to see what a country is leading with right now (~38 items per edition).

## `queries` (type: `array`):

Keyword searches, one per line (~100 items each). Google's operators work: quoted "exact phrase", OR, -exclude, site:reuters.com, intitle:, and time windows like when:1h / when:7d or after:2026-09-01 before:2026-09-05. Note allinurl: is NOT supported and returns nothing.

## `topics` (type: `array`):

Google's eight built-in news sections (50–70 items each). Case sensitive upstream — the actor validates them before sending.

## `locations` (type: `array`):

Local news by place name, e.g. Jakarta, London, San Francisco. An unrecognised place returns a stub feed upstream; the actor detects that and emits an ERROR row instead of a fake result.

## `topicIds` (type: `array`):

Raw Google News topic IDs (the CAAq... string in a /topics/ URL). Use these to reach sub-topics that have no named section, such as a specific sport or region.

## `publicationIds` (type: `array`):

Raw Google News publisher IDs (the string in a /publications/ URL) to pull a single outlet's feed.

## `editions` (type: `array`):

Country/language editions to fetch every selection from. Editions are genuinely different newsrooms, not translations. Denmark is deliberately absent: Google has no Danish edition and every Danish URL silently returns the Norwegian feed.

## `resolveArticleUrls` (type: `boolean`):

Turn Google's opaque redirect links into the publisher's real article URL. Strongly recommended — without it you get no clickable link, only the publisher's name and homepage. Costs one ~600 KB page fetch per article, so turn it off for fast headline-only sweeps.

## `maxItemsPerFeed` (type: `integer`):

Truncate each feed to this many items. 0 = keep everything the feed returns. This can only reduce results: Google News has no pagination, so a feed's own ceiling (~38 top / ~100 search / 50–70 topic) is the hard maximum for one request.

## `maxConcurrency` (type: `integer`):

Parallel article landing-page fetches. Only affects URL resolution; feeds are always fetched one at a time.

## `minRequestInterval` (type: `number`):

Minimum gap between request starts, in seconds. Paces starts without holding a worker, so it is the honest speed control once concurrency stops helping.

## `proxyConfiguration` (type: `object`):

Apify Proxy on the shared datacenter pool. This is the default because it is included in your plan at no extra cost and this target works through it. If you start seeing blocks, challenges or empty results, switch the group here to Residential -- it uses real consumer IPs and gets through more, but Apify bills residential traffic per gigabyte, so leave it off unless you need it.

## Actor input object example

```json
{
  "includeTopStories": false,
  "queries": [
    "artificial intelligence"
  ],
  "topics": [],
  "locations": [],
  "topicIds": [],
  "publicationIds": [],
  "editions": [
    "US:en"
  ],
  "resolveArticleUrls": true,
  "maxItemsPerFeed": 0,
  "maxConcurrency": 5,
  "minRequestInterval": 0.2,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `items` (type: `string`):

One row per scraped record. See the dataset's default view for field definitions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "artificial intelligence"
    ],
    "editions": [
        "US:en"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/google-news-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "queries": ["artificial intelligence"],
    "editions": ["US:en"],
}

# Run the Actor and wait for it to finish
run = client.actor("scrapyx/google-news-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "artificial intelligence"
  ],
  "editions": [
    "US:en"
  ]
}' |
apify call scrapyx/google-news-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/google-news-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/p9nvMCeRAfxjfq2vh/builds/P04IJ00DVZnjRq0qC/openapi.json