Google News Scraper

This Google News scraper turns search queries into clean full-text articles from the original publisher sites — text, images, author, date, and a quality score per article.

Google News → publisher URLs → clean article text + images + metadata, with JS rendering, paywall, and consent-page fallbacks. HTTP-first, Playwright only when needed.

Scrape one query or thousands in a single run. Each article lands as its own dataset row with the full text, images, author, source, language, and a quality score — ready for NLP pipelines, media monitoring, or research datasets.

What You Get

Each article in the output includes:

• title — headline as published
• url — canonical publisher URL (not the Google News redirect)
• source — publisher name (e.g. "Reuters", "TechCrunch")
• publishedAt — ISO 8601 timestamp
• author — byline when available
• text — clean full-text content (300+ characters, validated)
• images — OG image, featured image, and in-article images with alt text
• language — detected content language
• extractionSuccess — boolean flag for downstream filtering
• contentQuality — score (0-100), level (low/medium/high), and warnings

Set fetchArticleDetails: false to skip crawling and get RSS metadata only (title, source, date, link) at minimal cost.

Use Cases

• Media monitoring — track coverage of your company, product, or competitors across thousands of outlets from a single scheduled run.
• Brand & PR tracking — collect every article mentioning your brand as full text, ready for share-of-voice and sentiment analysis.
• NLP / LLM corpora — build clean full-text news datasets for training, fine-tuning, or RAG pipelines without writing per-site extractors.
• Financial news signals — feed trading research or alerting systems with timely full-text articles about tickers, sectors, and markets.
• Journalism research — assemble coverage timelines and source material for investigations and academic studies.

Quick Start

Using Apify Console

1. Visit Apify Console
2. Search for "Google News Scraper"
3. Configure your search parameters
4. Run the actor

Using Apify CLI

npm install -g apify-cli

# Single query
apify call xmolodtsov/google-news-scraper --input '{
  "query": "Tesla",
  "maxItemsPerUrl": 10
}'

# Multiple queries (string shorthand)
apify call xmolodtsov/google-news-scraper --input '{
  "queries": ["tesla", "apple"],
  "maxItemsPerUrl": 10
}'

# Multiple queries with passthrough fields
apify call xmolodtsov/google-news-scraper --input '{
  "queries": [
    { "query": "Kim Kardashian", "profileUrl": "https://news.google.com/search?q=kim+kardashian" },
    { "query": "MrBeast" }
  ],
  "maxItemsPerUrl": 10,
  "maxItems": 15
}'

Using Apify API (JavaScript)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });

const run = await client.actor('xmolodtsov/google-news-scraper').call({
    queries: [
        { query: 'Taylor Swift', profileUrl: 'https://news.google.com/search?q=taylor+swift' },
        { query: 'Elon Musk', profileUrl: 'https://news.google.com/search?q=elon+musk' },
    ],
    maxItemsPerUrl: 10,
    maxItems: 50,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
// items is a flat array of articles, each with query + passthrough fields merged in
console.log(items);

Using Apify API (Python)

from apify_client import ApifyClient

client = ApifyClient('YOUR_API_TOKEN')

run = client.actor('xmolodtsov/google-news-scraper').call(
    run_input={
        'queries': ['tesla', 'apple'],
        'maxItemsPerUrl': 10,
    }
)

for item in client.dataset(run['defaultDatasetId']).iterate_items():
    print(item['title'], item['url'])

Using the REST API

Run the actor and get the articles back in a single request:

curl -X POST "https://api.apify.com/v2/acts/xmolodtsov~google-news-scraper/run-sync-get-dataset-items?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "Tesla", "maxItemsPerUrl": 10}'

Input Modes

Most Common: Single Query

{
    "query": "artificial intelligence",
    "maxItemsPerUrl": 10
}

That's it — one query, up to 10 articles.

Bulk: Multiple Queries

Pass an array of strings to scrape several topics in one run:

{
    "queries": ["tesla", "apple", "nvidia"],
    "maxItemsPerUrl": 10
}

Advanced: Queries with Passthrough Fields

Each query can be an object. Any field besides query is passed through to every output article for that query — useful for linking results back to your own IDs, profile URLs, or tags:

{
    "queries": [
        { "query": "Kim Kardashian", "profileUrl": "https://news.google.com/search?q=kim+kardashian" },
        { "query": "MrBeast", "customField": "my-tag" },
        "Taylor Swift"
    ],
    "maxItemsPerUrl": 10,
    "maxItems": 25
}

Precedence: queries > query. If both are provided, queries wins.

Note on duplicates: deduplication runs per query, not across queries. Overlapping queries (e.g. "tesla" and "elon musk") can return the same article once for each query it matches. If you need global uniqueness, deduplicate downstream by url.

Configuration

Apify Proxy is required — leave it enabled. The actor uses Apify Proxy by default and Google News URL resolution depends on it. If you switch to custom proxies or set "useApifyProxy": false, resolution fails for all modern Google News URLs and the run returns 0 articles.

Input Parameters

Parameter	Type	Required	Default	Description
query	string	No*	-	Simple search query string
queries	array	No*	-	Array of strings or objects with query and optional passthrough fields
maxItemsPerUrl	integer	No	50	Max articles per individual query
maxItems	integer	No	0	Optional global cap on total articles (0 = unlimited)
fetchArticleDetails	boolean	No	true	If false, skip article crawling and return RSS metadata only
region	string	No	"US"	Country code (US, GB, CA, AU, DE, ES, MX, IT)
language	string	No	"en-US"	Language code (en-US, en-GB, en-CA, en-AU, de-DE, es-ES, es-MX, it-IT)
dateFrom	string	No	Last 30 days	Start date (YYYY-MM-DD). Only include articles published on or after this date; defaults to the last 30 days if not set
dateTo	string	No	Today	End date (YYYY-MM-DD). Only include articles published on or before this date; defaults to today if not set
topics	array	No	[]	Google News topic names to fetch (e.g. "business", "technology"). Used alongside or instead of a search query to browse topic feeds
topicsHashed	array	No	[]	Google News topic hashes for direct topic RSS feed URLs. Use this if you know the internal hash for a Google News topic section
disableBrowserFallback	boolean	No	false	Skip Playwright fallback — cheaper but may return fewer articles
proxyConfiguration	object	No	Apify Proxy enabled	Proxy settings; keep the default Apify Proxy (required — see callout above)

*At least one of query or queries is required.

How Extraction Works

The pipeline resolves every Google News redirect to the real publisher URL, then fetches and extracts the article in two separate layers:

Fetching — two transports, tried in order:

1. HTTP fetch — fast and cheap; used first for every article
2. Playwright browser — automatic fallback for articles that fail over HTTP (JS-rendered or consent-gated pages)

Extraction — six strategies applied to the fetched HTML, stopping at the first that yields 300+ characters of text with images:

1. Readability
2. Extractus
3. JSON-LD structured data
4. Custom per-site selectors
5. Meta tags
6. Heuristics

Every article is then quality-scored (text length, image presence, error-page detection). Low-quality results are filtered before they reach your dataset.

Why Fewer Articles Than Requested?

maxItemsPerUrl is a ceiling, not a guarantee. Before an article is saved it must pass three quality gates:

1. 300+ characters of extracted text
2. At least one valid image
3. Quality score of 25 or higher

Articles that fail any gate are dropped entirely. The crawler compensates: it keeps fetching additional candidates from the Google News RSS pool in batches until it either reaches your target count or exhausts the pool. For narrow queries, tight date ranges, or image-light publishers, the pool can run out first — and the run returns fewer articles than requested. That's expected behavior, not an error.

If you need every headline regardless of content quality, set fetchArticleDetails: false to get the raw RSS metadata for all discovered articles.

Estimated Cost

All costs depend on article count, target sites, and proxy tier. The numbers below are rough guidelines based on typical runs using Apify Proxy (datacenter tier).

Scenario	Articles	Typical Cost
RSS metadata only (fetchArticleDetails: false)	100	~$0.01 – $0.02
Full text, HTTP-first (most sites)	100	~$0.05 – $0.10
Full text, mixed HTTP + Playwright fallback	100	~$0.10 – $0.25
Heavy JS sites (frequent Playwright)	100	~$0.20 – $0.50

Cost levers you control:

• fetchArticleDetails: false — skip article crawling entirely for near-zero cost
• disableBrowserFallback: true — stay HTTP-only, ~2-5x cheaper, fewer articles from JS-heavy sites
• maxItemsPerUrl / maxItems — hard caps on article count
• Proxy tier — datacenter is default and cheapest; residential auto-escalates only on repeated 429/403 errors

Limitations

Be aware of these before you buy:

• Paywalled sites — articles behind hard paywalls (WSJ, FT, NYT subscriber-only) will return partial text or fail. The scraper extracts whatever is publicly visible.
• Heavy bot protection — sites with aggressive Cloudflare challenges or CAPTCHAs may need multiple retries and residential proxies, increasing cost.
• Region/language variance — Google News returns different articles depending on region and language. The same query may yield different results from US vs DE.
• RSS feed limits — Google News RSS feeds return a limited window of articles (roughly 24-72 hours). For historical coverage, use dateFrom/dateTo date slicing, which the scraper handles automatically.
• Image availability — some publishers strip images or serve them via CDN policies that block external access. Articles without valid images are skipped entirely (all-or-nothing policy); use fetchArticleDetails: false if you need every headline regardless of images.
• Duplicates across queries — deduplication is per query, so overlapping queries can return the same article more than once (see Input Modes).

Output Format

Output is a flat array of articles. Each article is a separate dataset entry with the query string and any passthrough fields merged at the top level:

[
    {
        "query": "Taylor Swift",
        "profileUrl": "https://news.google.com/search?q=taylor+swift",
        "title": "Taylor Swift Announces New Album - Billboard",
        "url": "https://www.billboard.com/2025/08/05/taylor-swift-new-album.html",
        "source": "Billboard",
        "publishedAt": "2025-08-05T14:08:57.000Z",
        "author": "Jane Smith",
        "text": "Full article content...",
        "description": "Brief summary of the article...",
        "images": [
            {
                "url": "https://example.com/image.jpg",
                "type": "featured-og",
                "alt": "Image description"
            }
        ],
        "tags": ["Taylor Swift"],
        "language": "en",
        "extractionSuccess": true,
        "contentQuality": {
            "score": 85,
            "level": "high",
            "isValid": true,
            "warnings": []
        }
    },
    {
        "query": "MrBeast",
        "customField": "test-passthrough",
        "title": "MrBeast Breaks YouTube Record",
        "url": "https://www.example.com/mrbeast-record.html",
        "source": "Example News",
        "publishedAt": "2025-08-05T10:00:00.000Z",
        "text": "Full article content...",
        "..."
    }
]

Integrations

• API & scheduling — run the actor via the Apify API, schedule recurring runs in Apify Console (e.g. hourly media monitoring), and attach webhooks to get notified when a run finishes.
• Export — download results from the dataset as JSON, CSV, Excel, XML, or RSS — directly in Console or via the API.
• Connect — plug the actor into Make, Zapier, or LangChain through Apify's native integrations to pipe fresh articles into your automations and LLM apps.

FAQ

Why did I get fewer articles than I asked for? Every saved article must pass strict quality gates, and Google's RSS pool for a query is finite. See Why Fewer Articles Than Requested? for the full explanation.

Can I get historical news? Google News RSS feeds only expose a rolling window of roughly 24-72 hours per feed. The scraper extends reach by slicing dateFrom/dateTo into multiple feed requests, but coverage is still bounded by what Google returns — deep archives are not available.

Does it work on paywalled articles? The scraper extracts whatever text is publicly visible. Hard paywalls (subscriber-only content) return partial text or fail; the actor does not bypass paywalls or log in anywhere.

How much does a run cost? See Estimated Cost. The biggest levers are fetchArticleDetails: false (near-zero cost), disableBrowserFallback: true (HTTP-only), and the maxItemsPerUrl/maxItems caps.

Why does the same article appear under two different queries? Deduplication runs per query. If two queries overlap (e.g. "tesla" and "elon musk"), an article matching both is returned once per query. Deduplicate downstream by url if you need global uniqueness.

What does the output look like with fetchArticleDetails: false? You get one row per RSS item with title, url (the Google News link, not the resolved publisher URL), source, publishedAt, the RSS description snippet as text, an empty images array, and extractionSuccess: false — no full text, no quality score. It's the cheapest way to get every headline.

Is It Legal to Scrape Google News?

This actor collects publicly available data only: it reads public Google News feeds and public article pages, requires no login, and does not bypass paywalls or other access controls. Scraping publicly accessible data is generally permitted, but you are responsible for how you use the results — make sure your use complies with applicable laws (copyright, GDPR and other data-protection rules) and the terms of the sites involved. This is not legal advice.

Development

Built with Crawlee. Local development and test instructions live in the GitHub repository.

License

MIT -- see LICENSE for details.

Acknowledgments

• Apify SDK and Crawlee for the scraping framework
• @mozilla/readability and @extractus/article-extractor for content extraction
• fast-xml-parser for RSS parsing