# 🌐 Google News Scraper — Publishers, Real URLs (`iettiu/news-aggregator`) Actor

Search world news across Google News and top publisher feeds in one run. Every competitor reads one source; this merges Google News, BBC, NYT, Guardian and your own RSS into one schema — with real decoded article URLs and cross-source dedupe. No API keys.

- **URL**: https://apify.com/iettiu/news-aggregator.md
- **Developed by:** [Dương Thiệu Trần](https://apify.com/iettiu) (community)
- **Categories:** News, Agents, MCP servers
- **Stats:** 1 total users, 0 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

from $2.00 / 1,000 article scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## 🌐 Global News Scraper — Google News + Publishers, One Schema

Search world news across **Google News and top publisher feeds in a single run**. Every other
news scraper on the Store reads one source; this one merges search-engine discovery with curated
publisher feeds and your own RSS, deduped across sources, into one clean schema — with the **real
article URLs**, not Google redirect links.

No API keys. No browser. No proxies. A default run finishes in a few seconds.

***

### Why scrape Google News and publisher RSS together

The Store is full of *Google News scrapers* — and they all return the same thing: Google's
redirect URLs, one source, no dedupe. This actor is built differently.

| | Typical Google News scraper | This actor |
|---|---|---|
| Sources per run | 1 (Google News) | **Google News + 9 publishers + your feeds** |
| Article URLs | `news.google.com/rss/articles/...` redirects | **real publisher URLs** (decoded) |
| Cross-source dedupe | none | ✅ same story from two sources collapses, keeping both |
| Languages | usually English | **en · vi · de · fr · es · ja** editions |
| Schema | one shape per actor | **one unified schema** across every source |

### News article data fields you get

```json
{
  "id": "a1b2c3d4e5f6",
  "title": "Tesla cuts EV prices across Europe",
  "url": "https://www.reuters.com/business/autos/tesla-...",
  "source": "Reuters",
  "source_type": "google_news",
  "url_decoded": true,
  "summary": "Tesla lowered prices on its Model 3 and Model Y across...",
  "language": "en",
  "published_at": "2026-08-14T09:12:00Z",
  "fetched_at": "2026-08-14T10:00:00Z",
  "also_seen_in": ["BBC"]
}
```

- **`url`** is the real publisher link, decoded from Google's redirect — competitors leave it as an
  unusable `news.google.com` URL.
- **`also_seen_in`** names every other source that carried the same story, so you keep the
  provenance instead of losing the duplicate.
- **`published_at`** is a proper timestamp — RFC-822 named-timezone dates (the standard RSS form)
  are parsed correctly, so nothing sorts to the bottom with a null date.

### News sources: Google News, BBC, NYT, Guardian

| Source | What it covers |
|---|---|
| **Google News** | Any topic you search, from hundreds of outlets, in six language editions |
| **Publisher feeds** | BBC · Guardian · NYT · TechCrunch · The Verge · Ars Technica · Wired · Al Jazeera · CNBC — mapped to their topic sections when you search technology/business/science |
| **Your feeds** | Paste any RSS/Atom URLs to add outlets no pack anticipates |

### Google News scraper inputs

| Input | What it does |
|---|---|
| `search` | Topic or keywords, e.g. `"electric cars"`. Empty = top headlines. |
| `sources` | Which sources: `google_news`, `publisher_rss`, `user_feeds`. |
| `locale` | Google News edition: US · UK · Vietnam · Germany · France · Spain/LatAm · Japan. |
| `feedUrls` | Your own RSS/Atom feed URLs. |
| `language` | Keep only one language, or `any`. |
| `excludeKeywords` | Drop articles containing these words, e.g. `"opinion, sponsored"`. |
| `publishedWithinDays` | Recency cutoff. Articles with no date are kept. |
| `maxItems` | Hard cap, so a run can't produce an unexpected bill. |

### Daily news brief — get only the new articles

Turn on **`onlyNew`** and schedule the Actor. The first run returns everything matching your
search; every run after that returns only what was published since. A topic you track every
morning becomes a short brief instead of the same headlines again.

Because you are billed per article returned, a daily brief costs a fraction of a full run.

| Run | With `onlyNew` off | With `onlyNew` on |
|---|---|---|
| Monday | 98 articles | 98 articles |
| Tuesday | mostly the same 98 | the 11 published overnight |
| Wednesday | mostly the same 98 | the 9 published since |

This also side-steps the ceiling described above: you are no longer trying to pull a large corpus
in one run, you are collecting it a day at a time.

History is kept on your own account for 30 days. If storage is ever unavailable the run returns
everything rather than silently returning nothing — an empty brief and a quiet news day look
identical, and only one of them is a bug.

#### Send it to Slack every morning

1. **Schedule** → new schedule, `0 8 * * *`, pointing at this Actor with `onlyNew: true`.
2. In n8n, Make or Zapier, add an **Apify → Actor run finished** trigger.
3. Map `title`, `source`, `url` and `published_at` into a Slack message.

`url` is already the real publisher link rather than a `news.google.com` redirect, so the message
links straight to the article.

### News Scraper pricing

Tiered by your Apify plan — **bigger plans pay less per article**:

| Your Apify plan | Per article |
|---|---|
| Free | $0.004 |
| Bronze | $0.0035 |
| Silver | $0.003 |
| **Gold / Platinum / Diamond** | **$0.002** |

Actor start: $0.00005 — effectively free.

| Run | Articles | Free plan | Gold+ |
|---|---|---|---|
| Default | 50 | **$0.20** | **$0.10** |
| Narrow search, everything available | ~100 | $0.40 | $0.20 |
| Top headlines, everything available | ~310 | $1.24 | $0.62 |

**You are charged per article returned, not per source read.** A run pulls from every selected
source and bills you only for what it delivers — so a query with little coverage costs little.

### How many articles can one run actually return?

`maxItems` is a **cap, not a target**. Setting it to 1,000 does not produce 1,000 articles; it
only stops the run from exceeding that. The real ceiling is set by the sources, and it is worth
knowing before you plan a job around it. Measured:

| Run | Articles returned |
|---|---|
| `search: "artificial intelligence"` | **~100** |
| `search: "technology"` (broad) | ~170 |
| no `search` — top headlines | **~310** |

Two limits stack:

- **Google News RSS returns at most 100 results per query.** That is Google's own cap. No input
  changes it, and it applies per query — not per run.
- **Publisher feeds are general news feeds.** They carry ~280 articles between them, but when you
  search a narrow topic, only the articles that happen to mention it survive. "artificial
  intelligence" matched **1** of 281; "technology" matched 84.

**To go beyond ~310, supply your own feeds.** `feedUrls` accepts up to 25 feeds at 200 items each,
so a run built on your own sources can reach several thousand articles. The built-in sources alone
cannot.

### What people build with news data

- **Media monitoring** — track a company, product or topic across many outlets at once.
- **Newsletter & digest automation** — feed a daily brief from Google News + your chosen feeds.
- **Research & sentiment datasets** — one normalised, deduped, multilingual news corpus.
- **LLM/RAG pipelines** — clean article metadata with real URLs, ready to enrich downstream.

### Google News scraper FAQ

#### Which news sources does this scraper cover?

Google News (any topic, six language editions), a curated pack of nine publishers — BBC, Guardian,
NYT, TechCrunch, The Verge, Ars Technica, Wired, Al Jazeera and CNBC — and any RSS or Atom feed you
supply yourself. All three are read in one run and merged into a single schema.

#### Does it return the real article URL or a Google redirect?

The real publisher URL. Google News hands out `news.google.com/rss/articles/CBMi...` redirect links
that are useless for downstream fetching; this Actor decodes them. If a decode fails it returns the
Google URL with `url_decoded: false` rather than guessing, so you always know which you have.

#### Can I scrape news in Vietnamese?

Yes — set `locale` to `vn` and `language` to `vi`. Language is detected from the script itself
rather than assumed from the locale, so Vietnamese diacritics aren't confused with French accents
and Japanese kana isn't misread as Chinese.

#### Can I use this for media monitoring?

That's the main use. Search a company, product or topic and get every outlet that covered it in one
pass, with `also_seen_in` naming the other sources carrying the same story so you can measure
pickup instead of counting the same article twice.

#### Does it return the full article text?

No — headlines, summaries, links and metadata only. Full article bodies are copyrighted, so
redistributing them isn't something this Actor does. The real URL is there if you have your own
licence to fetch the body.

#### Can I add my own RSS feeds?

Yes — pass them in `feedUrls`. They're read alongside the built-in sources and deduped against
them, so an outlet you add that also appears in Google News collapses into one row.

#### How much does it cost to scrape 1,000 news articles?

You're charged per article returned, not per source read. A run queries Google News plus every
publisher feed and bills only for what it delivers. See the pricing table above.

### Notes and limitations

- News articles are **public data**; this actor returns headlines, summaries and links, not full
  article bodies — no copyrighted text is redistributed.
- Google News redirect URLs are decoded to real publisher links where possible; if a decode
  fails, the row keeps the Google URL and `url_decoded: false` rather than guessing.
- One source failing degrades coverage, never the run — the log names what was skipped.

# Actor input Schema

## `search` (type: `string`):

Topic to search news for, e.g. "electric cars" or "vietnam economy". Leave empty for top headlines. Google News searches this directly; publisher feeds are filtered to match.

## `sources` (type: `array`):

Which sources to pull from. google\_news searches any topic; publisher\_rss is the curated outlet pack; user\_feeds reads the URLs you supply below.

## `locale` (type: `string`):

Which Google News regional edition to search (affects language and coverage).

## `feedUrls` (type: `array`):

Any RSS/Atom feed URLs to include (used when 'user\_feeds' is selected). One per line.

## `language` (type: `string`):

Keep only articles in this language. 'any' keeps all.

## `excludeKeywords` (type: `string`):

Drop articles whose title or summary contains any of these words, e.g. "opinion, sponsored".

## `publishedWithinDays` (type: `integer`):

Keep only articles published in the last N days. 0 keeps all. Articles with no date are kept.

## `onlyNew` (type: `boolean`):

Return only articles this Actor has not returned to you on a previous run. Turn this on when you schedule it: the first run gives you everything, and every run after that gives only what was published since. You are billed per article returned, so a daily brief costs a fraction of a full run. History is kept per account for 30 days; if it is ever unavailable the run returns everything rather than silently returning nothing.

## `maxItems` (type: `integer`):

Hard cap on how many articles are returned, so a run cannot produce an unexpected bill.

## `debugMode` (type: `boolean`):

Log the source mix and per-step counts.

## Actor input object example

```json
{
  "search": "artificial intelligence",
  "sources": [
    "google_news",
    "publisher_rss"
  ],
  "locale": "us",
  "feedUrls": [],
  "language": "any",
  "excludeKeywords": "",
  "publishedWithinDays": 0,
  "onlyNew": false,
  "maxItems": 50,
  "debugMode": false
}
```

# Actor output Schema

## `articles` (type: `string`):

One row per article. Unified schema across every source, with real (decoded) publisher URLs where available.

## `articlesCsv` (type: `string`):

The same articles as a CSV download.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "search": "artificial intelligence"
};

// Run the Actor and wait for it to finish
const run = await client.actor("iettiu/news-aggregator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "search": "artificial intelligence" }

# Run the Actor and wait for it to finish
run = client.actor("iettiu/news-aggregator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "search": "artificial intelligence"
}' |
apify call iettiu/news-aggregator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,iettiu/news-aggregator"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/L6Ksm070bYMfKGcnH/builds/KXaAUHLOvfSKdck5N/openapi.json