# Google News Scraper (`cirkit/google-news-scraper`) Actor Fast Google News scraper. Search by keyword with time-range filter, scrape top stories per country, topic sections, geo headlines, or topic IDs. Extracts title, publisher, date, snippet, image, related coverage. Optional decode of the wrapped Google URL into the real publisher URL. - **URL**: https://apify.com/cirkit/google-news-scraper.md - **Developed by:** [Crikit](https://apify.com/cirkit) (community) - **Categories:** News, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Google News Scraper Fast, reliable Google News scraper. Pulls news articles by keyword, country, topic section, local geo headlines, or direct Google News topic ID. Pure RSS read, no headless browser, no anti-bot stack. ### What it does Scrapes Google News for news articles and returns structured records with title, publisher, publish date, snippet, thumbnail, Google News URL, and clustered related coverage. Optionally resolves the wrapped Google News URL into the publisher's real article URL via Google's own batchexecute RPC. ### Inputs | Field | Type | Description | |---|---|---| | `queries` | string[] | Keywords or phrases. Supports Google operators: `site:reuters.com`, `intitle:`, `OR`, `-exclude`, `"exact phrase"`. | | `topics` | string[] | Editorial sections: `WORLD`, `NATION`, `BUSINESS`, `TECHNOLOGY`, `ENTERTAINMENT`, `SPORTS`, `SCIENCE`, `HEALTH`. | | `geos` | string[] | Free-text city names like `San Francisco`, `New York`, `Tokyo`. Google resolves to a geo topic ID. | | `topicIds` | string[] | Direct Google News topic ID (`CAAq...` base64). | | `includeTopStories` | boolean | Also scrape each locale's Top Stories feed. | | `locales` | object[] | Country editions. Each entry is `{hl, gl, ceid}`. Default: US English. Examples: `{hl: "fr-FR", gl: "FR", ceid: "FR:fr"}`, `{hl: "ja-JP", gl: "JP", ceid: "JP:ja"}`. | | `timeRange` | enum | Restrict keyword searches to a window: `1h`, `12h`, `1d`, `7d`, `30d`, `1y`. Use `any` for no restriction. | | `resolvePublisherUrls` | boolean | When true, decodes each Google News URL into the publisher's real URL. Adds one HTTP request per article. | | `maxResultsPerTarget` | integer | Cap per single feed (one query × one locale). | | `maxResults` | integer | Hard cap across the entire run. | | `maxConcurrency` | integer | Number of feeds fetched in parallel. | | `decodeConcurrency` | integer | Concurrent URL decoder calls. | | `proxyConfiguration` | object | Apify proxy settings. Defaults to RESIDENTIAL US. | You can mix any combination of `queries`, `topics`, `geos`, `topicIds`, and `includeTopStories` in a single run. ### Outputs Every record includes: - `articleId` - Google's stable CBM identifier for the article - `title` - article title (publisher suffix stripped) - `googleNewsUrl` - the wrapped Google News URL - `publisherUrl` - the publisher's real URL, when `resolvePublisherUrls=true` - `publisherName`, `publisherHomeUrl` - publication details - `publishedAt` - ISO-8601 UTC - `imageUrl` - thumbnail URL (note: Google News RSS rarely emits per-item images; this field is usually null) - `snippetHtml` - raw HTML snippet - `relatedCoverage[]` - other articles Google clustered under the same story - `feedKind`, `query`, `topic`, `topicId`, `geo`, `locale`, `scrapedAt` ### Example use cases - PR firms tracking brand mentions across publishers - Hedge funds running event-driven signal pipelines - Market intelligence teams monitoring competitor coverage - Sentiment analysis on a stream of topic-tagged news - Local-news dashboards by city - Country-by-country media coverage comparison ### Performance and pricing - Pure RSS reads: ~5 sec for 100 records on a single feed - No JS execution, no headless browser, datacenter proxy sufficient (RESIDENTIAL default for safety margin) - Pay per dataset item. Each successful record costs the same flat fee regardless of whether you used a query, topic, geo, or topic-id feed - The `resolvePublisherUrls` flag adds an extra HTTP request per article and slows the run roughly 3x. Disable it for the cheapest, fastest scrape ### Tips - For high-volume keywords, splice by `timeRange` (`1h`, `1d`, `7d`) and run the same query multiple times across windows. Each Google feed is hard-capped at 100 items - Fan a single query across multiple locales (`US:en`, `GB:en`, `FR:fr`, `JP:ja`) to multiply your coverage per run - Geo topic IDs are stable. The first time you scrape `San Francisco`, capture the topic ID from the run's `googleNewsUrl` and use `topicIds` directly in later runs to skip the redirect ### Stack Python + httpx + Apify SDK. No browser. Simple parser over Google's public RSS XML. # Actor input Schema ## `queries` (type: `array`): Keywords or phrases to search Google News for. Supports Google operators: site:reuters.com, intitle:, OR, -exclude, "exact phrase". ## `topics` (type: `array`): Editorial topic sections by name. Each scrapes the corresponding topic RSS feed. ## `geos` (type: `array`): Free-text city or place names for local headlines (e.g. 'San Francisco', 'New York', 'Tokyo'). Google maps the name to its internal geo topic ID. ## `topicIds` (type: `array`): Direct Google News topic IDs (base64 CAAq... strings from news.google.com URLs). Use when you want a specific Google News topic page. ## `includeTopStories` (type: `boolean`): Also scrape each locale's editorial Top Stories feed. ## `locales` (type: `array`): Country editions to scrape. Each locale fans out across every query/topic/geo input. Defaults to US English. ## `timeRange` (type: `string`): Restrict search results to a recency window. Only applies to keyword searches. Use 'any' (or leave blank) for no restriction. ## `resolvePublisherUrls` (type: `boolean`): When true, the actor decodes each Google News article URL into the publisher's real article URL (e.g. https://www.barrons.com/...). Adds one extra HTTP request per article. Disable for the cheapest, fastest run. ## `maxResultsPerTarget` (type: `integer`): Cap on items per single feed (one query in one locale, one topic in one locale, etc.). Google itself returns ~100 per feed; values above 200 are clamped. ## `maxResults` (type: `integer`): Hard cap across the entire run. Leave 0 or empty for unlimited. ## `maxConcurrency` (type: `integer`): Number of RSS feeds fetched in parallel. Higher values speed up multi-input runs. ## `decodeConcurrency` (type: `integer`): Number of publisher-URL decoder calls in parallel. Only relevant when resolvePublisherUrls=true. ## `proxyConfiguration` (type: `object`): Apify proxy configuration. Defaults to RESIDENTIAL US. Google News RSS has no anti-bot, but residential IPs avoid geo-routing surprises across locales. ## Actor input object example ```json { "queries": [ "tesla" ], "topics": [], "geos": [], "topicIds": [], "includeTopStories": false, "locales": [ { "hl": "en-US", "gl": "US", "ceid": "US:en" } ], "timeRange": "any", "resolvePublisherUrls": false, "maxResultsPerTarget": 50, "maxResults": 0, "maxConcurrency": 5, "decodeConcurrency": 3, "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ], "apifyProxyCountry": "US" } } ```` # Actor output Schema ## `dataset` (type: `string`): Dataset of Google News article records. ## `datasetCsv` (type: `string`): Same dataset rendered as CSV. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "queries": [ "tesla" ], "locales": [ { "hl": "en-US", "gl": "US", "ceid": "US:en" } ], "maxResultsPerTarget": 50, "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ], "apifyProxyCountry": "US" } }; // Run the Actor and wait for it to finish const run = await client.actor("cirkit/google-news-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "queries": ["tesla"], "locales": [{ "hl": "en-US", "gl": "US", "ceid": "US:en", }], "maxResultsPerTarget": 50, "proxyConfiguration": { "useApifyProxy": True, "apifyProxyGroups": ["RESIDENTIAL"], "apifyProxyCountry": "US", }, } # Run the Actor and wait for it to finish run = client.actor("cirkit/google-news-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "queries": [ "tesla" ], "locales": [ { "hl": "en-US", "gl": "US", "ceid": "US:en" } ], "maxResultsPerTarget": 50, "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ], "apifyProxyCountry": "US" } }' | apify call cirkit/google-news-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=cirkit/google-news-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Google News Scraper", "description": "Fast Google News scraper. Search by keyword with time-range filter, scrape top stories per country, topic sections, geo headlines, or topic IDs. Extracts title, publisher, date, snippet, image, related coverage. Optional decode of the wrapped Google URL into the real publisher URL.", "version": "0.5", "x-build-id": "O6lsvEyoKvxhyWFX3" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/cirkit~google-news-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-cirkit-google-news-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/cirkit~google-news-scraper/runs": { "post": { "operationId": "runs-sync-cirkit-google-news-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/cirkit~google-news-scraper/run-sync": { "post": { "operationId": "run-sync-cirkit-google-news-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "queries": { "title": "Search Queries", "uniqueItems": true, "type": "array", "description": "Keywords or phrases to search Google News for. Supports Google operators: site:reuters.com, intitle:, OR, -exclude, \"exact phrase\".", "default": [], "items": { "type": "string" } }, "topics": { "title": "Topic Sections", "uniqueItems": true, "type": "array", "description": "Editorial topic sections by name. Each scrapes the corresponding topic RSS feed.", "items": { "type": "string", "enum": [ "WORLD", "NATION", "BUSINESS", "TECHNOLOGY", "ENTERTAINMENT", "SPORTS", "SCIENCE", "HEALTH" ] }, "default": [] }, "geos": { "title": "Geographic Headlines", "uniqueItems": true, "type": "array", "description": "Free-text city or place names for local headlines (e.g. 'San Francisco', 'New York', 'Tokyo'). Google maps the name to its internal geo topic ID.", "default": [], "items": { "type": "string" } }, "topicIds": { "title": "Custom Topic IDs", "uniqueItems": true, "type": "array", "description": "Direct Google News topic IDs (base64 CAAq... strings from news.google.com URLs). Use when you want a specific Google News topic page.", "default": [], "items": { "type": "string" } }, "includeTopStories": { "title": "Include Top Stories", "type": "boolean", "description": "Also scrape each locale's editorial Top Stories feed.", "default": false }, "locales": { "title": "Locales", "type": "array", "description": "Country editions to scrape. Each locale fans out across every query/topic/geo input. Defaults to US English.", "default": [ { "hl": "en-US", "gl": "US", "ceid": "US:en" } ] }, "timeRange": { "title": "Time Range", "enum": [ "any", "1h", "12h", "1d", "7d", "30d", "1y" ], "type": "string", "description": "Restrict search results to a recency window. Only applies to keyword searches. Use 'any' (or leave blank) for no restriction.", "default": "any" }, "resolvePublisherUrls": { "title": "Resolve Publisher URLs", "type": "boolean", "description": "When true, the actor decodes each Google News article URL into the publisher's real article URL (e.g. https://www.barrons.com/...). Adds one extra HTTP request per article. Disable for the cheapest, fastest run.", "default": false }, "maxResultsPerTarget": { "title": "Max Results Per Feed", "minimum": 1, "maximum": 200, "type": "integer", "description": "Cap on items per single feed (one query in one locale, one topic in one locale, etc.). Google itself returns ~100 per feed; values above 200 are clamped.", "default": 100 }, "maxResults": { "title": "Global Max Results", "minimum": 0, "type": "integer", "description": "Hard cap across the entire run. Leave 0 or empty for unlimited.", "default": 0 }, "maxConcurrency": { "title": "Feed Fetch Concurrency", "minimum": 1, "maximum": 20, "type": "integer", "description": "Number of RSS feeds fetched in parallel. Higher values speed up multi-input runs.", "default": 5 }, "decodeConcurrency": { "title": "URL Decoder Concurrency", "minimum": 1, "maximum": 10, "type": "integer", "description": "Number of publisher-URL decoder calls in parallel. Only relevant when resolvePublisherUrls=true.", "default": 3 }, "proxyConfiguration": { "title": "Proxy Configuration", "type": "object", "description": "Apify proxy configuration. Defaults to RESIDENTIAL US. Google News RSS has no anti-bot, but residential IPs avoid geo-routing surprises across locales.", "default": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ], "apifyProxyCountry": "US" } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```