# Google News Scraper: Search, Topics, Decoded URLs (`automation_craft/google-news-scraper`) Actor

Google News search, topic, place, publisher and top headline feeds in any country edition, no API key. Day by day windows go past the 100 article cap. Returns headline, publisher, publication time, snippet and the decoded publisher URL as JSON, CSV or Excel. Pay once per new article.

- **URL**: https://apify.com/automation\_craft/google-news-scraper.md
- **Developed by:** [Automation Craft](https://apify.com/automation_craft) (community)
- **Categories:** News, Marketing, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.80 / 1,000 articles

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

### Google News Scraper: Search, Topics, Decoded URLs

**Scrape Google News** without an API key or a login. Search with Google's own operators, read topic, place,
publisher and top headline feeds in any country edition, get the real publisher URL instead of the Google
redirect link, and go past Google's 100 article limit by splitting your date range day by day.

Pure HTTP, no browser, no proxy. A run of 20 articles with decoded URLs finishes in a few seconds on 256 MB.
You pay per new article, and a named memory means a daily monitor never pays for the same article twice.

### Quick start

1. Put your search terms in **Search terms, topics, places or Google News URLs**, one per line. Operators
   work: `after:2026-09-01`, `before:2026-09-30`, `when:7d`, `site:bbc.com`, `source:Reuters`,
   `"exact phrase"`, `-excluded`. You can also paste a `news.google.com` search, topic, section or place URL.
2. Choose the **Feed type**. Search is the default. Topic takes `TECHNOLOGY`, `BUSINESS`, `SPORTS` and the
   rest, or a topic id. Place takes a city, region or country name. Publisher takes `bbc.com` or `Reuters`.
3. Set **Country edition** and **Language** if you do not want the United States English edition.
4. Set **Last N days** or an explicit **Date from** and **Date to**, then leave **Deep search** on.
5. Set **Max articles in total**, press Start, and download the dataset as JSON, CSV or Excel.

To turn a run into a monitor, fill in **Dedup memory name**, save the run as a task and schedule it. Every
later run delivers only articles that memory has not seen, and the known ones cost nothing.

### What you get

One row per article. The fill rates below were measured on 2026-09-15 and the raw measurement is committed in the repository at
`research/fill-rates-2026-09-15.json`, not rounded up from memory: 300 live articles across three queries and 221 publishers,
plus a 60 article topic feed, a 100 article search feed and a 45 article body sample.

| Field | What it is | Fill rate |
|---|---|---|
| `title` | Headline, with the trailing publisher name removed | 100% |
| `url` | The publisher's own article URL when decoding succeeded, otherwise the Google News link | 100% |
| `googleUrl` | The original news.google.com link, always present | 100% |
| `urlDecoded` | Whether `url` is the publisher URL or still the Google link | 100% decoded |
| `sourceName` | Publisher name as Google News reports it | 100% |
| `sourceUrl`, `sourceDomain` | Publisher home page and its host | 100% |
| `publishedAt` | Publication time, ISO 8601 UTC | 100% |
| `snippet` | The feed's summary line. Google repeats the headline here for essentially every item, so treat it as a headline, not a summary | 100% present, 100% equal to the headline |
| `query`, `feedType`, `shardWindow` | Which query, feed and date window produced the row | 100% |
| `country`, `language`, `locale` | The Google News edition the row was read from | 100% |
| `isNew` | False only for articles your dedup memory already knew | 100% |
| `relatedCoverage` | Other articles Google clustered with this one | 54 of 60 rows on a topic feed, 0 of 100 on a search feed |
| `body`, `wordCount`, `paywalled`, `bodyRetrieved`, `bodyStatus` | Publisher article text, only when you ask for it | 32 of 45 pages returned a body, median 564 words |
| `bodySourceUrl` | The page the text was really read from, after any publisher redirect | present on every row that asked for a body |

Every run that completes writes one `run-summary` row with per query counters: articles found, delivered, filtered,
dropped as outside the date range, deduplicated in the run, known and skipped, feed windows requested,
windows that came back truncated, feed errors, decode successes and failures, and body successes and
failures. It also reports what the run actually charged, split into what was requested and what the platform
confirmed.

### Going past the 100 article limit

One Google News feed request returns about 100 items, whatever you ask for. There is no page or offset
parameter. Deep search splits your date range into one request per day, and splits the still open day into
rolling hour windows when that day is still full.

Measured on 2026-09-15 for the query "artificial intelligence": one seven day feed returned **100** articles,
seven one day windows returned **694** unique articles. Google News is a live feed, so the exact number moves
between runs; repeated measurements the same day ranged from 496 to 696 against an unchanging 100.

When a window still comes back at the limit, the run summary counts it in `cappedShards` and the Actor says
so in plain text. It does not pretend the window was complete.

### How much does it cost to scrape Google News?

Pay per event. A run that finds nothing pays only Apify's standard start fee of $0.00005 and no per article
charge at all.

| Event | Free and Bronze | Silver | Gold and above |
|---|---|---|---|
| **Article** | $0.001 | $0.0009 | $0.0008 |
| **Article body** | $0.002 | $0.0018 | $0.0016 |
| **Actor start** | $0.00005 | $0.00005 | $0.00005 |

Apify bills the start event once per gigabyte of run memory. This Actor runs on 256 MB by default, so it is
one start event per run unless you raise the memory yourself.

An Article is charged once, when a new unique article is delivered. Decoding the publisher URL is included in
that price. An Article body is charged only when the publisher's text was actually delivered, at least 100
words. Duplicates, filtered rows, articles your dedup memory already knows, warning rows and the run summary
are all free. So are pages that turn out to be blocked, empty, a cookie consent screen, a subscription
teaser, or a different story from the one the link promised.

A page the publisher marks as not free to read but still publishes the whole text is delivered, flagged
`paywalled: true`, and charged like any other body: you got the article. A page that only offers a
subscription teaser is rejected and costs nothing. Nothing is ever unlocked or bypassed.

So 1,000 new articles with decoded URLs cost **$1.00005** on the Bronze tier, or **$0.80005** on Gold: the
articles plus one start event. Adding article bodies to all 1,000 costs up to $2.00 more, and less in
practice, because a page that is blocked, empty, a subscription teaser or a story we cannot identify is not charged. The Actor start fee is billed
per gigabyte of memory and this Actor runs on 256 MB, so it is one event per run.

### Input

Every field has a plain English title and description in the Console. The ones that change what you pay:

- **Max feed requests per query** caps how many requests one query may make, hour windows included. Days
  beyond the cap are not requested, and the run summary counts them.
- **Max articles per query** and **Max articles in total** cap the charged rows. The total is shared fairly
  between your queries, then any unused budget goes back to the queries that still have articles waiting.
- **Decode publisher URLs** is on by default in the Console. It is included in the Article price.
- **Include the article body** is off unless you switch it on, including for API callers, so nobody is
  silently upcharged.
- **Dedup memory name** makes known articles free. **Re-emit known articles** adds them back as free rows
  flagged `isNew: false`, and those free rows carry feed metadata only. They have their own budget, so a run
  with the option on can return up to twice **Max articles in total** rows. The charged part is never more
  than **Max articles in total**, whatever the split between new and known turns out to be.
- **Only these publishers** and **Never these publishers** accept domains or names. For search terms you
  type, the include list is pushed into the query so Google filters server side; for a pasted Google News
  URL, a topic, a place or top headlines, both lists are applied to what comes back. Either way a filtered
  out article is never charged.

An empty input is valid: the Actor returns the country's top headlines, capped at 20 articles.

The Console prefills and the values an API caller gets when they leave a field out are deliberately
different. Omit them from an API call and you get: **Decode publisher URLs** off, **Include the article
body** off, **Max articles per query** 100 and **Max articles in total** 1,000. Nothing that costs extra is
ever switched on for a caller who did not ask for it.

### What this Actor does NOT do

- It does not bypass paywalls or consent walls. A page it cannot read is delivered with `bodyRetrieved:
  false` and a `bodyStatus` that says why, and it is not charged.
- It does not scrape the Google News web app or the Google Search news tab. Feeds come from the public RSS
  endpoints; decoding a link additionally loads that article's own Google News page and asks Google's link
  resolver for the publisher URL; article bodies come from the publisher's page.
- Google News publication pages (`news.google.com/publications/...`) and Full coverage story pages
  (`/stories/...`) have no RSS feed. Pasting one returns a free warning row telling you to use the publisher
  domain instead. Publisher filtering by domain or name works.
- Google's `after:` and `before:` operators accept dates, not times, so a past day cannot be split into
  hours. Only the day that is still open can, using rolling `when:Nh` windows.
- It does not return article images or author names.
- `snippet` is the feed's own summary line, and Google fills it with the headline again for almost every
  item. If you need a real summary, take the article body.
- It is not a Google Search scraper, and it does not read the Google Search News tab.
- **Two runs of the same dedup memory started at the same time can both deliver the same article.** The
  memory is a key value record, not a lock. Schedule a memory's runs so they do not overlap.
- If the platform refuses a charge, the row is still delivered and the article is remembered, so it is
  never billed later. The run summary reports what was requested and what the platform confirmed, and they
  can differ.
- Article bodies are read from wherever the publisher's link ends up after redirects, which is reported in
  `bodySourceUrl`. The cookie consent detector is tuned on English wording; on other languages a consent
  page is usually still rejected for being too short, but not always.
- One feed request can return an article a few hours outside the day it was asked for. Rows outside the
  requested range are dropped, counted in the summary as `outOfWindow`, and never charged.
- A query that returns nothing new for twelve windows in a row stops, and a run whose first forty windows
  return no article at all stops. Both are reported in the run summary. On a very sparse historical range
  this can end a query before the oldest days, so narrow the query rather than widening the range.
- If the dedup memory you named cannot be opened, the run delivers the articles and charges nothing for
  them. If it stops saving part way through, everything from that point on is free. Articles delivered and
  saved before the failure are charged normally, because they are safely remembered. It is not possible to
  promise "pay once per article" for rows the memory cannot record, so those rows are free instead.
- Article bodies are only taken from a page that agrees it is the article we asked for. A page whose
  structured data names a different story returns `bodyStatus: identity-mismatch`; a page whose markup
  carries several stories the Actor cannot tell apart returns `bodyStatus: ambiguous-page`. Both deliver no
  body and cost nothing. Where a publisher labels nothing at all, the single substantial story on the page
  is used, so this is a strong filter rather than a guarantee.

### Use it from the API

```bash
curl -X POST "https://api.apify.com/v2/acts/automation_craft~google-news-scraper/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"queries":["artificial intelligence"],"lastNDays":7,"maxTotal":200,"maxArticlesPerQuery":200,"decodeUrls":true}'
```

```javascript
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('automation_craft/google-news-scraper').call({
    queries: ['site:techcrunch.com'],
    lastNDays: 30,
    maxTotal: 500,
    maxArticlesPerQuery: 500,
    decodeUrls: true,
    memoryName: 'techcrunch-monitor',
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
```

```python
from apify_client import ApifyClient
client = ApifyClient("YOUR_TOKEN")
run = client.actor("automation_craft/google-news-scraper").call(run_input={
    "queries": ["TECHNOLOGY"],
    "feedType": "topic",
    "country": "GB",
    "language": "en-GB",
    "maxTotal": 100,
    "maxArticlesPerQuery": 100,
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item.get("title"), item.get("url"))
```

### FAQ

#### Can I scrape Google News without the official API?

Yes. Google retired its News API years ago, and the public RSS feeds are what is left. This Actor reads those
feeds, so there is no API key, no Google account and no login anywhere in the flow.

#### How do I get more than 100 articles from Google News?

Leave Deep search on and give the run a date range. One feed request is capped at about 100 items and has no
page or offset parameter, so the only way through is narrower windows. The Actor requests one window per day
and deduplicates across them. In a live test, seven day windows returned 694 unique articles where a single
seven day feed returned 100.

#### How do I get the real article URL instead of the Google News redirect link?

Leave **Decode publisher URLs** on. The Actor resolves each Google News link to the publisher's own URL and
puts it in `url`, keeping the original in `googleUrl`. When Google refuses to resolve a link, the row is
still delivered with `urlDecoded: false` and the Google link, and it is not billed differently.

#### Can I get the full article text?

Switch on **Include the article body**. The Actor fetches the publisher's page and extracts the article
text, a word count and a paywall flag. In a 45 article sample taken on 2026-09-15, 32 pages returned a body
with a median of 564 words; the other 13 were refused by the publisher, failed to connect, offered only a
subscription teaser, carried several stories the Actor could not tell apart, or were too short to be worth
charging for. None of those were charged. The full breakdown is in `research/fill-rates-2026-09-15.json`.

#### How do I search Google News for older articles or a fixed date range?

Use **Date from** and **Date to**, or **Last N days**. On search and publisher feeds they are translated
into Google's `after:` and `before:` operators, and Deep search then walks the range one day at a time. Topic,
place and top headline feeds have no date operators at all, so for those the range is applied to each
article's publication time instead, and the run summary says so. Google indexes older articles unevenly, so a
range far in the past returns fewer rows per day than a recent one.

#### Why does this Actor run with limited permissions?

Least privilege. It only needs its own storages, so that is all it asks for. The one shared thing it touches
is the cross run dedup memory, which is a named key value store it creates in your account on first use and
reads back on later runs.

### More data tools by Automation Craft

- [Google Trends Scraper - Compare and Trending Now](https://apify.com/automation_craft/google-trends-scraper)
- [Substack Scraper: Posts, Notes and Profiles](https://apify.com/automation_craft/substack-scraper)
- [Telegram Channel Scraper: Posts, Views, Dates](https://apify.com/automation_craft/telegram-channel-scraper)
- [Meta Ad Library Scraper - All Placements, Filters](https://apify.com/automation_craft/meta-ads-library-scraper)
- [G2 Reviews Scraper: Ratings, Pros and Cons](https://apify.com/automation_craft/g2-reviews-scraper)
- [LinkedIn Jobs Scraper - No Login, Real Dates](https://apify.com/automation_craft/linkedin-jobs-scraper)
- [Amazon Product Scraper - Search, Best Sellers](https://apify.com/automation_craft/amazon-data-scraper)

### Changelog

**1.0** (2026-09-15): first release. Search, topic, section, place, publisher and top headline feeds; pasted
Google News URLs; day and hour window sharding past the 100 item limit; publisher URL decoding; optional
article bodies with a paywall flag; named cross run dedup memory; per query and per run caps; run summary
with per query counters.

# Actor input Schema

## `queries` (type: `array`):

What to collect. In Search mode these are keywords and may carry Google News operators: after:2026-09-01, before:2026-09-30, when:7d, site:bbc.com, source:Reuters, "exact phrase", -excluded. You can also paste a news.google.com search, topic, section or geo URL of any kind and it is parsed into the right feed. Leave empty to get the country's top headlines.

## `feedType` (type: `string`):

Which Google News surface the entries above refer to. Search runs a keyword search. Topic takes one of Google's topic names or a topic id. Geo takes a city, region or country name. Publisher takes a domain such as bbc.com or a publisher name such as Reuters. Section feeds are addressed by pasting their URL. Top headlines needs no entries at all.

## `country` (type: `string`):

Google News edition to read, sent as the gl parameter. Changes which outlets and which regional stories appear.

## `language` (type: `string`):

Interface and results language, sent as the hl parameter.

## `lastNDays` (type: `integer`):

Collect articles published in the last N days, counting today. Ignored when Date from or Date to is set. Topic, place and top headline feeds carry no date operators, so for those the range is applied to each article's publication time instead of to the request.

## `dateFrom` (type: `string`):

First publication date to collect, inclusive. Overrides Last N days. Google News accepts dates, not times.

## `dateTo` (type: `string`):

Last publication date to collect, inclusive. Overrides Last N days.

## `deepSearch` (type: `boolean`):

On: one feed request per day in the range, plus rolling hour windows for today when a day is still full, so a query can return far more than Google's 100 item limit. Off: one request covering the whole range. Only Search and Publisher feeds can carry Google's date operators.

## `maxShardsPerQuery` (type: `integer`):

Safety cap on how many feed requests one query may make, hour windows included. Days beyond the cap are not requested and are counted in the run summary. Raise it for long back catalogues.

## `maxArticlesPerQuery` (type: `integer`):

Cap on charged articles for each entry in the list above. Honoured exactly.

## `maxTotal` (type: `integer`):

Hard cap on charged articles for the whole run, across every query. Free re-emitted articles do not count against it.

## `decodeUrls` (type: `boolean`):

Resolve each Google News link to the real publisher URL. Included in the Article price, no extra event. When Google cannot resolve a link the row is still delivered with the Google News link and urlDecoded set to false.

## `includeArticleBody` (type: `boolean`):

Fetch each publisher page and extract the article text, word count and a paywall flag. Charged as Article body only when a body of at least 100 words was delivered. Requires Decode publisher URLs. Cookie consent pages and subscription teasers are rejected, and so is a page whose own markup names a different story or carries several stories the Actor cannot tell apart. Where a publisher labels nothing, the single substantial story on the page is used. No paywall is ever bypassed.

## `memoryName` (type: `string`):

Give this schedule a name and the Actor remembers every article it has already delivered under that name. Later runs skip those articles for free, so a daily monitor pays once per article. Leave empty to dedup inside the run only.

## `reEmitKnown` (type: `boolean`):

Deliver articles the memory already knows as extra rows flagged isNew false, at no charge. They do not count against Max articles in total. Those free rows carry feed metadata only: no publisher URL decoding and no article body.

## `resetMemory` (type: `boolean`):

Forget everything stored under the memory name and start again. Every article found in this run is then new and charged.

## `includeSources` (type: `array`):

Keep only articles from these publishers. Domains (bbc.com) and names (Reuters) both work. For search terms you type, the list is pushed into the query so Google filters server side; for a pasted Google News URL, a topic, a place or top headlines, it is applied to what comes back. Either way a filtered out article is never charged.

## `excludeSources` (type: `array`):

Drop articles from these publishers. Domains or names.

## `excludeWords` (type: `array`):

Drop an article when its headline or snippet contains any of these words, case insensitive.

## `concurrency` (type: `integer`):

How many Google News requests run at once. Eight is measured as fast and clean; lower it if you run many scrapers from one account.

## `proxyConfiguration` (type: `object`):

Optional. Google News answers Apify run containers directly, so this Actor runs with no proxy by default. Switch one on only if you need a specific egress country.

## Actor input object example

```json
{
  "queries": [
    "artificial intelligence"
  ],
  "feedType": "search",
  "country": "US",
  "language": "en-US",
  "lastNDays": 7,
  "deepSearch": true,
  "maxShardsPerQuery": 60,
  "maxArticlesPerQuery": 20,
  "maxTotal": 20,
  "decodeUrls": true,
  "concurrency": 8
}
```

# Actor output Schema

## `items` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "artificial intelligence"
    ],
    "feedType": "search",
    "country": "US",
    "language": "en-US",
    "lastNDays": 7,
    "deepSearch": true,
    "maxShardsPerQuery": 60,
    "maxArticlesPerQuery": 20,
    "maxTotal": 20,
    "decodeUrls": true,
    "includeArticleBody": false,
    "concurrency": 8
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation_craft/google-news-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "queries": ["artificial intelligence"],
    "feedType": "search",
    "country": "US",
    "language": "en-US",
    "lastNDays": 7,
    "deepSearch": True,
    "maxShardsPerQuery": 60,
    "maxArticlesPerQuery": 20,
    "maxTotal": 20,
    "decodeUrls": True,
    "includeArticleBody": False,
    "concurrency": 8,
}

# Run the Actor and wait for it to finish
run = client.actor("automation_craft/google-news-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "artificial intelligence"
  ],
  "feedType": "search",
  "country": "US",
  "language": "en-US",
  "lastNDays": 7,
  "deepSearch": true,
  "maxShardsPerQuery": 60,
  "maxArticlesPerQuery": 20,
  "maxTotal": 20,
  "decodeUrls": true,
  "includeArticleBody": false,
  "concurrency": 8
}' |
apify call automation_craft/google-news-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation_craft/google-news-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/vhBBgIZbeA0e2Vi9g/builds/zo1IydkcZHs3HA9wE/openapi.json