# Google News Scraper — Real URLs, Full Text & Monitoring (`studioploski/google-news-scraper`) Actor

Scrape Google News by keyword or topic. Real publisher URLs, past the 100-result limit, only-new mode for scheduled monitoring, optional full article text. Plain HTTP, pay only for saved articles.

- **URL**: https://apify.com/studioploski/google-news-scraper.md
- **Developed by:** [Luke P](https://apify.com/studioploski) (community)
- **Categories:** News, Marketing, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Google News Scraper — real publisher URLs, monitoring & full text

Scrape **Google News** by keyword or topic feed. Each item includes the headline, publisher, publish time, snippet, related coverage, the Google News link, and — when enabled — the **real publisher URL** and optional **full article text**.

Plain HTTP (no browser). Works with Google News RSS, decodes article IDs through Google’s batchexecute endpoint, and can page past the usual ~100-result cap by slicing the date window day by day.

### Why this scraper

- **Real publisher URLs** — not only `news.google.com` redirect links
- **Past the 100-result limit** — day slicing with `after:` / `before:` when you ask for more than 100 items or a date range
- **Only-new monitoring** — remember seen article IDs for scheduled runs; pay only for new items
- **Optional full text** — JSON-LD `articleBody` or `<article>` paragraphs from the publisher page
- **Pay per result** — charge only for articles (and full texts) actually saved

### Pricing

- **$1.50 per 1,000 articles** ($0.0015 each)
- **+$1.50 per 1,000 full texts** when full-text extraction succeeds

No monthly rental. Set a max total charge on the run; the actor stops cleanly and keeps everything already saved.

### Input

| Field | What it does | Default |
|---|---|---|
| `queries` | Keywords; Google operators allowed (`when:1d`, `site:cnbc.com`, …) | — |
| `topics` | `TOP_STORIES`, `WORLD`, `NATION`, `BUSINESS`, `TECHNOLOGY`, `ENTERTAINMENT`, `SPORTS`, `SCIENCE`, `HEALTH` | — |
| `language` / `country` | Google `hl` / `gl` (ceid derived automatically) | en-US / US |
| `timeRange` | `any`, `1h`, `1d`, `7d`, `30d`, `1y` → `when:` on queries | 7d |
| `dateFrom` / `dateTo` | `YYYY-MM-DD`; override timeRange | — |
| `maxArticlesPerQuery` | Cap per query/topic (1–5000). Over 100 triggers day slicing | 100 |
| `decodeUrls` | Resolve real publisher URLs | true |
| `fetchFullText` | Download publisher page body | false |
| `onlyNew` + `monitorKey` | Skip IDs stored under this monitor name | false / default |
| `proxyConfiguration` | Optional proxy; retries rotate session | none |
| `requestTimeoutSecs` | Per-request timeout | 30 |

#### Example — one-off search

```json
{
  "queries": ["artificial intelligence", "site:reuters.com climate"],
  "timeRange": "7d",
  "maxArticlesPerQuery": 50,
  "decodeUrls": true,
  "fetchFullText": false
}
```

#### Example — scheduled monitoring (only new)

```json
{
  "queries": ["tesla"],
  "timeRange": "1d",
  "maxArticlesPerQuery": 100,
  "onlyNew": true,
  "monitorKey": "tesla-daily",
  "decodeUrls": true
}
```

Run on a schedule. The first run saves matches; later runs skip IDs already stored in the `google-news-monitor` key-value store under `seen-tesla-daily` (up to 20 000 recent IDs).

#### Example — topic feed + full text

```json
{
  "topics": ["TECHNOLOGY"],
  "country": "US",
  "language": "en-US",
  "maxArticlesPerQuery": 30,
  "fetchFullText": true
}
```

### Output (one item per article)

```json
{
  "position": 1,
  "title": "Tesla’s self-driving Robotaxi service expands into greater Houston",
  "publisher": "Houston Public Media",
  "publisherUrl": "https://www.houstonpublicmedia.org",
  "url": "https://www.houstonpublicmedia.org/articles/news/transportation/2026/10/26/565273/teslas-robotaxi-houston-service-expands/",
  "googleNewsUrl": "https://news.google.com/rss/articles/CBMi…",
  "urlDecoded": true,
  "publishedAt": "2026-10-26T18:04:00.000Z",
  "publishedTimestamp": 1761501840000,
  "snippet": null,
  "relatedCoverage": [],
  "source": { "type": "query", "value": "tesla" },
  "language": "en-US",
  "country": "US",
  "fullText": "Optional article body…",
  "author": "Jane Doe",
  "image": "https://…",
  "scrapedAt": "2026-10-27T16:00:00.000Z"
}
```

`RUN_SUMMARY` in the default key-value store lists per-source found/saved/decoded counts and errors.

### FAQ

**Why is `url` null?** Decoding failed for that item (layout change, block, or timeout). `googleNewsUrl` is still present and `urlDecoded` is false. The run continues.

**Why fewer than `maxArticlesPerQuery`?** Google may not have that many matching stories in the window, or day slices returned duplicates (deduped by article ID).

**Does `onlyNew` cost anything for duplicates?** No. Seen IDs are skipped before push/charge.

**Full text empty?** Many sites block bots or ship body text only via JavaScript. You still get metadata; `fullTextError` explains the failure. Full-text events are charged only when text was obtained.

### Limitations

- Google News RSS returns at most ~100 items per request; higher limits rely on day slicing (day is the smallest slice).
- Publisher full-text quality varies; paywalled or JS-heavy pages often yield no body.
- Decode and full-text need outbound HTTP; use a proxy if your environment is blocked.
- Not affiliated with Google. Public news listings only; no login or personal data.

# Actor input Schema

## `queries` (type: `array`):

Keywords or Google News operators (when:1d, after:YYYY-MM-DD, site:example.com, …).

## `topics` (type: `array`):

Optional Google News topic feeds (in addition to queries).

## `language` (type: `string`):

Google News language, e.g. en-US, fr, de, ja.

## `country` (type: `string`):

Two-letter country/region code, e.g. US, CA, GB.

## `timeRange` (type: `string`):

Appended as when: for queries that do not already set when:/after:/before:. Ignored when dateFrom/dateTo are set.

## `dateFrom` (type: `string`):

YYYY-MM-DD. Overrides timeRange when set (with dateTo).

## `dateTo` (type: `string`):

YYYY-MM-DD. Overrides timeRange when set (with dateFrom).

## `maxArticlesPerQuery` (type: `integer`):

When over 100 (or a date range is given and a feed is full), the window is sliced by day to go past Google's ~100-item RSS cap.

## `decodeUrls` (type: `boolean`):

Resolve news.google.com article links to the real publisher URL.

## `fetchFullText` (type: `boolean`):

GET the publisher page and extract article body (JSON-LD or <article>/<p>), author and image.

## `onlyNew` (type: `boolean`):

Skip article IDs already seen under monitorKey (for scheduled monitoring runs).

## `monitorKey` (type: `string`):

Named seen-ID set in the google-news-monitor key-value store (sanitized).

## `requestTimeoutSecs` (type: `integer`):

How long to wait for each HTTP response before retrying.

## `proxyConfiguration` (type: `object`):

Optional. Each retry uses a fresh session/IP when a proxy is set.

## Actor input object example

```json
{
  "queries": [
    "artificial intelligence"
  ],
  "language": "en-US",
  "country": "US",
  "timeRange": "7d",
  "maxArticlesPerQuery": 100,
  "decodeUrls": true,
  "fetchFullText": false,
  "onlyNew": false,
  "monitorKey": "default",
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {}
}
```

# Actor output Schema

## `articles` (type: `string`):

All scraped articles in the default dataset (overview table view).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "artificial intelligence"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("studioploski/google-news-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "queries": ["artificial intelligence"] }

# Run the Actor and wait for it to finish
run = client.actor("studioploski/google-news-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "artificial intelligence"
  ]
}' |
apify call studioploski/google-news-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,studioploski/google-news-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/a2KUj10EeO0hLGll5/builds/8hVz9YRW2vpmvqF1Z/openapi.json