# 📰 Extract Google News Articles — AI & RAG Ready (`muhammadafzal/google-news-scraper`) Actor Extract Google News articles by keyword, topic, or URL with full-text extraction for AI/RAG pipelines. Get headlines, sources, snippets, images, authors, and clean article text in structured JSON. Export scraped data, run the scraper via API, or integrate with other tools. - **URL**: https://apify.com/muhammadafzal/google-news-scraper.md - **Developed by:** [Muhammad Afzal](https://apify.com/muhammadafzal) (community) - **Categories:** News, AI, MCP servers - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing from $7.00 / 1,000 results This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## 📰 Google News Scraper — AI & RAG Ready Extract Google News articles by keyword, topic, or URL with full-text extraction optimized for AI and RAG pipelines. Get headlines, sources, snippets, images, authors, and clean article text in structured JSON. Perfect for sentiment analysis, brand monitoring, competitive intelligence, and media tracking. ### Features - **Multi-input mode** — Search by keyword, browse by topic, or provide direct Google News URLs - **Full-text extraction** — Visit article pages and extract clean text without ads, navigation, or boilerplate - **AI/RAG-ready output** — Semantic field names, structured metadata, word count, and reading time estimates - **25+ countries & 17 languages** — Localized results for global news coverage - **Time filtering** — Filter by last hour, day, week, month, or year - **URL decoding** — Automatically resolves Google News redirect URLs to original article links - **RSS-first approach** — Uses Google News RSS feeds for reliable, fast data extraction - **Article metadata** — Extracts Open Graph tags, author, keywords, and description from publisher pages ### Use Cases - **RAG pipelines & LLM training** — Feed clean article text into vector databases and language models - **Brand monitoring & PR tracking** — Track mentions across thousands of news sources in real time - **Sentiment analysis** — Collect structured news data for NLP and sentiment models - **Competitive intelligence** — Monitor competitors, market trends, and industry developments - **Media aggregation** — Build custom news feeds combining topics, keywords, and regions - **Content curation** — Discover trending stories and curate content for newsletters and blogs ### Input | Parameter | Type | Default | Description | |---|---|---|---| | **searchQueries** | Array of strings | `["technology"]` | Keyword searches. Supports Google operators (OR, -, "exact", site:). | | **topics** | Array of strings | `[]` | News topics: WORLD, NATION, BUSINESS, TECHNOLOGY, ENTERTAINMENT, SPORTS, SCIENCE, HEALTH | | **startUrls** | Array of URLs | `[]` | Direct Google News URLs to scrape | | **maxResults** | Integer | `50` | Maximum articles per search query or topic (1–5000) | | **country** | String | `US` | Country code for localized results (25+ countries) | | **language** | String | `en` | Language code (17 languages supported) | | **timePeriod** | String | `1d` | Time filter: `1h`, `1d`, `7d`, `30d`, `1y`, `all` | | **extractFullText** | Boolean | `false` | Extract full article text from publisher pages (slower, essential for AI/RAG) | | **decodeUrls** | Boolean | `true` | Decode Google redirect URLs to original article links | | **proxyConfiguration** | Object | Apify Proxy | Proxy settings for scraping | ### Output Each article contains: | Field | Type | Description | |---|---|---| | `title` | String | Article headline | | `url` | String \| null | Decoded original article URL (when decodeUrls enabled) | | `source` | String \| null | Publisher name (e.g., "BBC News", "Reuters") | | `publisher_url` | String \| null | Publisher website URL (e.g., "https://www.reuters.com") | | `author` | String \| null | Article author (when extractFullText enabled) | | `published_at` | String \| null | Publication timestamp (ISO 8601) | | `snippet` | String \| null | Article summary from Google News | | `image_url` | String \| null | Article thumbnail image URL | | `category` | String \| null | News category (e.g., "TECHNOLOGY", "BUSINESS") | | `topic` | String \| null | Google News topic when browsing by topic | | `search_query` | String \| null | The search query that produced this result | | `full_text` | String \| null | Full article text (when extractFullText enabled) | | `word_count` | Integer \| null | Word count of extracted text | | `estimated_reading_time_min` | Integer \| null | Estimated reading time in minutes | | `article_metadata` | Object \| null | Open Graph and meta tags from the article page | | `scraped_at` | String | ISO 8601 extraction timestamp | | `source_url` | String | Google News feed URL that produced this result | | `publisher_url` | String \| null | Publisher website URL (e.g., "https://www.reuters.com") | ### Pricing This actor uses Pay-Per-Event pricing: | Event | Price | Description | |---|---|---| | `result` | $0.003 | Per article with headline, source, snippet, and metadata | | `full-text-result` | $0.010 | Per article with full text, author, word count, and reading time | **Examples:** - 100 articles (headlines only) = $0.30 - 100 articles (with full text) = $1.00 - 1,000 articles (headlines only) = $3.00 - 1,000 articles (with full text) = $10.00 ### Tips & Limitations - Start without `extractFullText` for fast headline-only results, then enable it when you need article body text - Google News RSS feeds return up to ~100 articles per search query; use multiple queries or topics for broader coverage - Time period filtering only applies to keyword searches, not topic pages - For best results with AI/RAG pipelines, enable `extractFullText` to get clean article text - The `decodeUrls` option resolves Google News redirect URLs — keep it enabled for original article links - Use Apify Proxy with residential rotation if you encounter rate limiting ### Integration Examples #### API ```bash curl -X POST "https://api.apify.com/v2/acts/USERNAME~google-news-scraper/runs?token=YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{"searchQueries": ["AI technology", "climate change"], "maxResults": 50, "extractFullText": true}' ```` #### Schedule Set up scheduled runs via Apify Console for continuous news monitoring and brand tracking. #### MCP Integration This actor is optimized for AI agents via the Apify MCP server. Semantic field names and rich dataset schemas make it easy for LLMs to understand and use the output. # Actor input Schema ## `searchQueries` (type: `array`): List of keyword searches to run on Google News. Supports Google search operators: use OR for multiple terms, minus sign to exclude words, quotes for exact match, site: to filter by source. Use this field when the user wants to find news about a topic, brand, person, or event. Do NOT use this when the user provides a direct URL — use startUrls instead. ## `topics` (type: `array`): Select Google News topic pages to browse. Use this when the user wants to browse top headlines by category, not search for a specific keyword. ## `startUrls` (type: `array`): Direct Google News URLs to scrape. Use this field when the user provides a specific Google News URL. Do NOT use this when the user describes a topic or wants to search by keyword — use searchQueries or topics instead. ## `maxResults` (type: `integer`): Maximum number of news articles to extract per search query or topic. Each article counts as one result for billing. Lower values complete faster. ## `country` (type: `string`): Country code for localized Google News results. Affects which regional edition of Google News is scraped. ## `language` (type: `string`): Language code for Google News results. Affects the interface language and may affect which articles are shown. ## `timePeriod` (type: `string`): Filter news articles by publication date. Only applies to keyword searches — topic pages show their own curated results. ## `extractFullText` (type: `boolean`): Visit each article URL and extract the full text content. Essential for AI/RAG pipelines and sentiment analysis. Increases runtime and cost per result. When disabled, returns headline, snippet, source, and metadata only. ## `decodeUrls` (type: `boolean`): Resolve Google News redirect URLs to the original article URLs. When enabled, each article URL is decoded to show the real publisher link instead of the Google redirect. Recommended for most use cases. ## `proxyConfiguration` (type: `object`): Select proxy to use for scraping. Datacenter proxies work well for Google News RSS feeds. Switch to residential proxies if you encounter blocks. ## Actor input object example ```json { "searchQueries": [ "technology" ], "topics": [], "startUrls": [], "maxResults": 50, "country": "US", "language": "en", "timePeriod": "1d", "extractFullText": false, "decodeUrls": true, "proxyConfiguration": { "useApifyProxy": true } } ``` # Actor output Schema ## `results` (type: `string`): Array of news article objects with headlines, sources, snippets, and optional full text. Ready for RAG pipelines, sentiment analysis, and media monitoring. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "searchQueries": [ "technology" ] }; // Run the Actor and wait for it to finish const run = await client.actor("muhammadafzal/google-news-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "searchQueries": ["technology"] } # Run the Actor and wait for it to finish run = client.actor("muhammadafzal/google-news-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "searchQueries": [ "technology" ] }' | apify call muhammadafzal/google-news-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=muhammadafzal/google-news-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "📰 Extract Google News Articles — AI & RAG Ready", "description": "Extract Google News articles by keyword, topic, or URL with full-text extraction for AI/RAG pipelines. Get headlines, sources, snippets, images, authors, and clean article text in structured JSON. Export scraped data, run the scraper via API, or integrate with other tools.", "version": "1.0", "x-build-id": "vCCbrXUaKwela4j3o" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/muhammadafzal~google-news-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-muhammadafzal-google-news-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/muhammadafzal~google-news-scraper/runs": { "post": { "operationId": "runs-sync-muhammadafzal-google-news-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/muhammadafzal~google-news-scraper/run-sync": { "post": { "operationId": "run-sync-muhammadafzal-google-news-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "searchQueries": { "title": "Search Queries", "type": "array", "description": "List of keyword searches to run on Google News. Supports Google search operators: use OR for multiple terms, minus sign to exclude words, quotes for exact match, site: to filter by source. Use this field when the user wants to find news about a topic, brand, person, or event. Do NOT use this when the user provides a direct URL — use startUrls instead.", "default": [ "technology" ], "items": { "type": "string" } }, "topics": { "title": "News Topics", "type": "array", "description": "Select Google News topic pages to browse. Use this when the user wants to browse top headlines by category, not search for a specific keyword.", "items": { "type": "string", "enum": [ "WORLD", "NATION", "BUSINESS", "TECHNOLOGY", "ENTERTAINMENT", "SPORTS", "SCIENCE", "HEALTH" ], "enumTitles": [ "🌍 World", "🏛️ Nation", "💼 Business", "💻 Technology", "🎬 Entertainment", "⚽ Sports", "🔬 Science", "🏥 Health" ] }, "default": [] }, "startUrls": { "title": "Start URLs", "type": "array", "description": "Direct Google News URLs to scrape. Use this field when the user provides a specific Google News URL. Do NOT use this when the user describes a topic or wants to search by keyword — use searchQueries or topics instead.", "default": [], "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "maxResults": { "title": "Max Results", "minimum": 1, "maximum": 5000, "type": "integer", "description": "Maximum number of news articles to extract per search query or topic. Each article counts as one result for billing. Lower values complete faster.", "default": 50 }, "country": { "title": "Country", "enum": [ "US", "GB", "CA", "AU", "IN", "DE", "FR", "JP", "BR", "IT", "ES", "MX", "KR", "RU", "CN", "SA", "AE", "ZA", "NG", "AR", "NL", "PL", "TR", "SE", "NO" ], "type": "string", "description": "Country code for localized Google News results. Affects which regional edition of Google News is scraped.", "default": "US" }, "language": { "title": "Language", "enum": [ "en", "es", "fr", "de", "pt", "it", "ja", "ko", "zh-CN", "ar", "hi", "ru", "nl", "pl", "tr", "sv", "no" ], "type": "string", "description": "Language code for Google News results. Affects the interface language and may affect which articles are shown.", "default": "en" }, "timePeriod": { "title": "Time Period", "enum": [ "1h", "1d", "7d", "30d", "1y", "all" ], "type": "string", "description": "Filter news articles by publication date. Only applies to keyword searches — topic pages show their own curated results.", "default": "1d" }, "extractFullText": { "title": "Extract Full Article Text", "type": "boolean", "description": "Visit each article URL and extract the full text content. Essential for AI/RAG pipelines and sentiment analysis. Increases runtime and cost per result. When disabled, returns headline, snippet, source, and metadata only.", "default": false }, "decodeUrls": { "title": "Decode Google News URLs", "type": "boolean", "description": "Resolve Google News redirect URLs to the original article URLs. When enabled, each article URL is decoded to show the real publisher link instead of the Google redirect. Recommended for most use cases.", "default": true }, "proxyConfiguration": { "title": "Proxy Configuration", "type": "object", "description": "Select proxy to use for scraping. Datacenter proxies work well for Google News RSS feeds. Switch to residential proxies if you encounter blocks.", "default": { "useApifyProxy": true } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```