# Google News Scraper: Keyword & Company News Alerts (`impressionable_lupine/google-news-scraper`) Actor

Scrape Google News for any keyword, company or site: query. Get publisher URLs (not Google redirects), dates, full article text, and only-new alerts across runs. $2 per 1,000 articles, no proxy needed.

- **URL**: https://apify.com/impressionable_lupine/google-news-scraper.md
- **Developed by:** [Majd Hijazin](https://apify.com/impressionable_lupine) (community)
- **Categories:** News, Automation
- **Stats:** 2 total users, 1 monthly users, 75.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 article scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Google News Scraper — keyword & company news alerts with full text

**Google News Scraper** turns any Google News search into clean, structured JSON: headline, publisher, **real publisher article URL** (not a `news.google.com` redirect), publication date, snippet and — optionally — the **full article text**. Turn on **only-new mode** and schedule it to get **company news alerts** that never repeat an article you already received.

It reads the public **Google News RSS** feed (`news.google.com/rss/search`), so it is fast (under a second per query for headlines), needs **no proxies and no browser** at moderate volume, and costs **$2 per 1,000 articles**.

- 🔎 **Any query**: keywords, `"exact company names"`, `site:techcrunch.com`, `intitle:`, `OR`, `-exclude`
- 🌍 **Any edition**: language (`hl`), country (`gl`) and edition (`ceid`) — e.g. `de` / `DE` / `DE:de`
- ⏱️ **Time range**: past hour, 24 hours, 7 days, 30 days, year — enforced on every article's date
- 🔗 **Publisher URLs resolved** from Google's encoded links without a browser
- 📰 **Full text** via Mozilla Readability — respects `robots.txt`, skips paywalls gracefully, and you only pay when extraction succeeds
- 🔔 **Only new articles** across runs (named key-value store), ideal for daily/hourly alerts
- ✅ **Publication-date check**: compares Google's date with the publisher's own `datePublished`

### What data do you get?

| Field | Example |
|---|---|
| `query` | `site:techcrunch.com startup funding` |
| `title` | `Otter.ai raises $320 million from General Catalyst` |
| `publisher` / `publisherUrl` | `TechCrunch` / `https://techcrunch.com` |
| `articleUrl` | `https://techcrunch.com/2026/10/20/otter-ai-raises-320-million-...` |
| `urlResolved` | `true` (false = still a Google link) |
| `googleNewsUrl` | `https://news.google.com/rss/articles/CBMi...` |
| `publishedAt` | `2026-10-20T14:02:11.000Z` (UTC) |
| `publisherPublishedAt` / `publishedAtCheck` | `2026-10-20T14:01:00.000Z` / `verified` |
| `snippet` | Publisher description (when available) |
| `fullText` / `wordCount` / `fullTextStatus` | Article body / `582` / `ok` |
| `author`, `imageUrl`, `relatedCoverage` | When the publisher declares them |
| `language`, `country`, `scrapedAt` | `en-US`, `US`, run time |

Each run also writes a summary to the key-value store record `OUTPUT`: per-status counts, URL-resolution stats, HTTP stats, query errors and the billing total.

### Examples

**1. Company news alerts (schedule daily, pay only for new articles)**

```json
{ "queries": ["\"OpenAI\"", "\"Anthropic\"", "\"Mistral AI\""], "timeRange": "1d", "onlyNew": true, "maxArticlesPerQuery": 50 }
```

**2. Keyword monitoring with full article text for LLM summarisation / RAG**

```json
{ "queries": ["AI regulation EU", "chip export controls"], "timeRange": "7d", "fetchFullText": true, "maxArticlesPerQuery": 20 }
```

**3. One publication only (site: operator)**

```json
{ "queries": ["site:techcrunch.com startup funding"], "timeRange": "7d" }
```

**4. German edition**

```json
{ "queries": ["Elektroauto Batterie"], "language": "de", "country": "DE", "timeRange": "30d" }
```

### Pricing

Pay-per-event, platform usage included:

| Event | Price | When |
|---|---|---|
| `article-scraped` | **$0.002** ($2 / 1,000) | Every article returned in the dataset |
| `article-fulltext` | **$0.003** | Only when the full text was extracted successfully (`fullTextStatus = ok`) |

- 1,000 headlines with resolved URLs: **$2.00**
- 1,000 articles with full text (70–85% extracted successfully in our tests): ≈ **$4.10–$4.55**
- Paywalled, robots-blocked or failed extractions are **not** charged the full-text fee.
- With `onlyNew`, already-seen articles are skipped **before** charging.
- The run never exceeds your **Max total charge (USD)**: budget is reserved before each article and full-text fetch, and the run stops cleanly when it is reached.

### Use via API

```bash
curl -X POST "https://api.apify.com/v2/acts/<username>~google-news-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"queries":["\"Nvidia\""],"timeRange":"1d","maxArticlesPerQuery":20}'
```

JavaScript:

```js
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('<username>/google-news-scraper').call({ queries: ['"Stripe"'], timeRange: '7d', fetchFullText: true });
const { items } = await client.dataset(run.defaultDatasetId).listItems();
```

### Use from AI agents (MCP)

Add the Apify MCP server to Claude Desktop, Cursor or any MCP client and expose this Actor as a tool:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?actors=<username>/google-news-scraper",
      "headers": { "Authorization": "Bearer <APIFY_TOKEN>" }
    }
  }
}
```

Then ask: *"Find this week's news about Stripe and summarise the three most important articles."* The agent calls the Actor with `queries`, `timeRange` and `fetchFullText`, and gets structured JSON back.

### How it works

1. Fetches `https://news.google.com/rss/search?q=<query> when:<range>&hl=..&gl=..&ceid=..`.
2. Parses the RSS (no DTD/entity expansion), strips the "– Publisher" suffix from titles, keeps Google's rank.
3. Resolves each Google link to the publisher URL: legacy link IDs decode offline; current opaque IDs are resolved with Google's own public `batchexecute` call (1 page fetch + 1 batched POST, no browser, no login).
4. Optional full text: checks `robots.txt` for the publisher, fetches the page once, extracts with Mozilla Readability, reads JSON-LD/meta for `datePublished`, author and image, and detects paywalls (`isAccessibleForFree: false`, HTTP 401/402, "subscribe to continue" teasers, known hard-paywall domains).
5. Dedupes within the run (same story across queries) and, with `onlyNew`, against previous runs.

### Limitations (honest)

- **~100 articles per query.** Google News RSS returns at most about 100 items per search (often fewer for niche queries). Split broad topics into several narrower queries or use shorter time ranges and schedules.
- **Rate limits.** Google requests are serialized with a 400 ms gap and retried with back-off on 429/5xx. In local tests (three back-to-back runs, ~50 Google requests each, 5 queries × 8 articles) we saw **zero 429s**. Heavy bursts from one IP can trigger Google's "unusual traffic" page; the Actor then stops resolving URLs for the rest of the run (articles keep their Google link, `urlResolved: false`) instead of hammering. We do not use proxies or solve CAPTCHAs. Comfortable volume: a few hundred queries and a few thousand articles per hour per run.
- **Operator queries can come back empty.** We observed Google intermittently returning an empty feed for queries containing `when:` / `site:`. The Actor retries once without the operators and enforces them itself (time window on `pubDate`, `site:` on the publisher domain); the run summary counts these as `operatorFallbacks`.
- **URL resolution costs 2 Google requests per article** (~120 KB). Disable `resolvePublisherUrls` for maximum speed if Google links are enough.
- **Snippets.** Google's RSS has no real article summary for most items; `snippet` is filled from the publisher's meta description when full text is enabled, otherwise often `null`.
- **Full text is best-effort.** In our tests 70–85% of articles extracted (`ok`). The rest are reported with a reason: `paywalled`, `robots_disallowed` (e.g. Reuters, Politico disallow crawlers), `blocked` (Cloudflare/403), `too_short`, `fetch_failed`. JavaScript-only pages are not rendered.
- **Dates** come from Google's `pubDate`. `publishedAtCheck = mismatch` flags pages whose own date differs by more than 36 h (republished or evergreen pages).
- **Terms.** Google states the RSS feed is intended for personal feed readers. You are responsible for how you use the data and for complying with Google's and publishers' terms and copyright. Store and redistribute full texts only where you have the right to.

### FAQ

**Is there an official Google News API?** No. Google retired its News API; the RSS feed is the public, stable interface, which is what this Actor uses.

**Do I need proxies?** Not at moderate volume. The Actor runs without proxies by design and paces its requests.

**How do news alerts work?** Set `onlyNew: true` and schedule the Actor (e.g. hourly). Seen articles are remembered for 120 days in the named key-value store `google-news-scraper-state` under a key derived from your queries (or your own `stateKey`). Only new articles are returned — and charged.

**Why is `articleUrl` sometimes a news.google.com link?** Resolution failed or was disabled (`urlResolved: false`). Full text is only attempted for resolved URLs.

**Can I get articles older than a year?** Use `timeRange: "any"` with specific queries; Google still caps each feed at ~100 items.

**What languages are supported?** Any Google News edition: set `language` (hl), `country` (gl) and optionally `ceid`.

**Will I be charged for paywalled articles?** Only the $0.002 article event (you still get the headline, URL and date). The $0.003 full-text fee applies only when text was extracted.

**How fast is it?** Headlines only (URL resolution off): 25 articles in under 1 s. With URL resolution: ~0.8 s per article (Google pacing). With full text, 5 queries × 8 articles took ~40 s end-to-end in our local test.

# Actor input Schema

## `queries` (type: `array`):

One Google News search per line: keywords (<code>AI regulation</code>), company names (<code>"Stripe"</code>), or Google search operators such as <code>site:techcrunch.com</code>, <code>intitle:</code>, <code>OR</code>, <code>-exclude</code>. Each query returns up to 100 articles (Google's RSS limit).

## `timeRange` (type: `string`):

Only return articles published within this window. Sent to Google as <code>when:</code> and also enforced on each article's publication date.

## `maxArticlesPerQuery` (type: `integer`):

Stop after this many articles for each query (1–100). Google News RSS returns at most ~100 per query.

## `fetchFullText` (type: `boolean`):

Visit each publisher page and extract the readable article body (Mozilla Readability). Respects robots.txt and skips paywalled pages. Adds $0.003 per article only when text is extracted successfully.

## `onlyNew` (type: `boolean`):

Skip articles already returned by a previous run with the same queries (or the same State key). Ideal for scheduled news alerts: you only pay for new articles.

## `language` (type: `string`):

Google News interface language, e.g. <code>en-US</code>, <code>en-GB</code>, <code>de</code>, <code>fr</code>, <code>es-419</code>, <code>ja</code>.

## `country` (type: `string`):

Two-letter edition country, e.g. <code>US</code>, <code>GB</code>, <code>DE</code>, <code>IN</code>.

## `ceid` (type: `string`):

Advanced. Defaults to <code>COUNTRY:lang</code> (e.g. <code>US:en</code>, <code>DE:de</code>). Override only for editions like <code>IN:hi</code>.

## `resolvePublisherUrls` (type: `boolean`):

Convert Google News redirect links to the real publisher article URL (no browser needed). Turn off for maximum speed; articleUrl then stays a news.google.com link.

## `stateKey` (type: `string`):

Optional name for the dedupe memory. Use the same key across tasks to share 'seen' history; leave empty to derive it from the queries + language.

## `skipPaywalledDomains` (type: `boolean`):

Do not fetch full text from known hard-paywall publishers (WSJ, FT, Bloomberg, NYT…). The article row is still returned with fullTextStatus = paywalled.

## `maxFullTextChars` (type: `integer`):

Truncate fullText to this many characters.

## Actor input object example

```json
{
  "queries": [
    "OpenAI",
    "\"Nvidia\" earnings",
    "site:techcrunch.com startup funding"
  ],
  "timeRange": "7d",
  "maxArticlesPerQuery": 10,
  "fetchFullText": false,
  "onlyNew": false,
  "language": "en-US",
  "country": "US",
  "resolvePublisherUrls": true,
  "skipPaywalledDomains": true,
  "maxFullTextChars": 50000
}
```

# Actor output Schema

## `articles` (type: `string`):

One row per article: query, title, publisher, articleUrl, publishedAt, snippet, fullText and status fields.

## `fullText` (type: `string`):

No description

## `runSummary` (type: `string`):

Counts, URL-resolution stats, full-text statuses, billing summary and per-query errors.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "OpenAI",
        "\"Nvidia\" earnings",
        "site:techcrunch.com startup funding"
    ],
    "maxArticlesPerQuery": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("impressionable_lupine/google-news-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "queries": [
        "OpenAI",
        "\"Nvidia\" earnings",
        "site:techcrunch.com startup funding",
    ],
    "maxArticlesPerQuery": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("impressionable_lupine/google-news-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "OpenAI",
    "\\"Nvidia\\" earnings",
    "site:techcrunch.com startup funding"
  ],
  "maxArticlesPerQuery": 10
}' |
apify call impressionable_lupine/google-news-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,impressionable_lupine/google-news-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ih4CoOkTsCqEd1uuu/builds/NfLClKYfTI3VM4kWB/openapi.json