# AI News Content Crawler (`ilborso/ai-news-content-crawler`) Actor This Apify actor searches Google News RSS for one or more country/language editions, collects article URLs, opens each news page, extracts clean text, and stores one dataset item per processed URL. It uses Google News RSS discovery with automatic HTTP fallback when browser rendering fails. - **URL**: https://apify.com/ilborso/ai-news-content-crawler.md - **Developed by:** [Fabio Borsotti](https://apify.com/ilborso) (community) - **Categories:** AI, Automation, Integrations - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing from $1.00 / 1,000 results This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Google News RSS Article Text Extractor This Apify actor searches Google News RSS for one or more country/language editions, collects article URLs, opens each news page, extracts clean text, and stores one dataset item per processed URL. It uses Google News RSS discovery plus Playwright-based extraction with automatic HTTP fallback when browser rendering fails. ### What it does - Accepts a Google News search query, including advanced operators such as `intitle`, `inurl`, `site`, `-`, quoted exact matches, `AND`, and `OR`. - Accepts one or more `languageAndRegion` pairs in the format `CC:ll`, for example `IT:it`, `US:en`, `LT:lt`, and uses them to build Google News RSS parameters `gl`, `hl`, and `ceid`. - Searches Google News RSS for each selected edition and decodes Google News article links into publisher URLs using `googlenewsdecoder`. - Limits discovery to a maximum number of unique news sites per language/region pair through `maxSites`. - Uses `daysBack` as the RSS search window and sets `chunkDays = daysBack` in the final actor logic. - Opens discovered article URLs with Playwright, extracts main page text, and falls back to HTTP + BeautifulSoup text extraction if Playwright fails. - Builds an `Accept-Language` header from the selected edition language and includes English as fallback for non-English editions. - Processes multiple pages in parallel using `maxConcurrency`, capped to 50 open Playwright pages at a time. - Writes one dataset item per processed URL and emits detailed performance logs for RSS discovery, URL decoding, browser extraction, fallback extraction, and overall actor runtime. ### Input #### Input fields | Field | Type | Required | Description | |---|---|---|---| | `searchQuery` | string | Yes | Query written as for the Google News search bar. Advanced operators are supported. Enter search query as you would write it to Google News search bar. You can use advanced operators such as intitle, inurl, site, exclude operator -, exact match with double-quotes, AND, OR and more. | | `maxConcurrency` | integer | No | Maximum number of pages opened simultaneously in the Playwright browser. Capped to 50. This value represents the maximum number of browser pages or tabs processed in parallel, not the number of system threads. Max value is capped to 50. | | `languageAndRegion` | array of strings | Yes | One or more country/language pairs in the format `CC:ll`, for example `IT:it`, `US:en`, `LT:lt`. If you specify more then one pair foreach pair will be returned maxSites site back. Practical whitelist of commonly used Google News country/language pairs in the format CC:ll. This is not an official or complete list of all combinations supported by Google News. | | `maxSites` | integer | No | Maximum number of unique news sites returned for each language/region pair. RSS feed finds more results, but domain deduplication filter stops when it reaches maxSites unique domains. | | `daysBack` | integer | No | Number of days back to search in Google News RSS. How many days back to search. chunkDays is always set equal to daysBack. | | `decodeInterval` | integer | No | Interval passed to `googlenewsdecoder`. Default is `1`. This is used to time the decoding of Google News links: it controls how often the decoder attempts to resolve the URL before considering it failed or moving on to the next attempt. In your case, a value of 1 essentially indicates a very short interval between attempts, useful for not overloading the service and reducing the risk of errors or rate limits. Increasing the value slows the process but can be more stable when decoding many links in sequence. | #### Example input ```json { "searchQuery": "\"samsung galaxy s25\" -ultra", "maxConcurrency": 10, "languageAndRegion": ["IT:it", "US:en", "LT:lt"], "maxSites": 50, "daysBack": 7, "decodeInterval": 1 } ```` ### How it works The actor runs in two stages. First, it queries Google News RSS for each selected language/region pair, applies the `after:` and `before:` filters derived from `daysBack`, decodes Google News links into publisher URLs, and deduplicates the results. Second, it opens each discovered news URL with Playwright, attempts to extract the main text from article-like selectors such as `article`, `main`, and common content containers, and falls back to a plain HTTP request plus HTML text cleanup when browser extraction fails. ### Concurrency The actor uses a shared Playwright browser and limits concurrent page work with `asyncio.Semaphore(maxConcurrency)`, so `maxConcurrency` represents the maximum number of browser pages or tabs processed in parallel, not the number of OS threads. This makes concurrency easy to reason about operationally: if `maxConcurrency` is set to `10`, the actor will process up to ten pages at the same time in Chromium. ### Language handling For each `languageAndRegion` pair, the actor extracts the language part and builds the `Accept-Language` header used for both Playwright and HTTP fallback requests. For non-English editions it uses a language list such as `["it", "en"]`, which becomes a header like `it,en;q=0.9`; for English editions it uses `["en"]`. This improves localization hints to publisher sites, but the final language of the returned page still depends on how each site handles geographic and language negotiation. ### Output Each dataset item contains the processing result for one discovered publisher URL. #### Successful Playwright result ```json { "url": "https://example.com", "success": true, "statusCode": 200, "langs": ["it", "en"], "acceptLanguage": "it,en;q=0.9", "htmlLang": "en", "text": "Example Domain ...", "error": null, "extractor": "playwright" } ``` #### Successful fallback result ```json { "url": "https://example.com", "success": true, "statusCode": 200, "langs": ["it", "en"], "acceptLanguage": "it,en;q=0.9", "htmlLang": "en", "text": "Example Domain ...", "error": "Playwright failed: ...", "extractor": "httpx-fallback" } ``` #### Final failure result ```json { "url": "https://example.com", "success": false, "statusCode": null, "langs": ["it", "en"], "acceptLanguage": "it,en;q=0.9", "htmlLang": null, "text": null, "error": "Playwright: ... | HTTP fallback: ...", "extractor": null } ``` ### Logs and performance metrics The final actor includes structured logs that make it easier to debug and profile runs. Key events include actor lifecycle logs such as `actor.start`, `actor.input`, and `actor.done`; RSS discovery logs such as `rss.collect.start`, `rss.chunk.start`, `rss.collect.stop`; extraction logs such as `playwright.start`, `extract.playwright_failed`, `extract.batch.start`, and `extract.batch.progress`; and timing logs such as `timing.parse_feed`, `timing.decode_google_news_url`, `timing.playwright_extract`, `timing.httpx_fallback`, `timing.discovery_total`, `timing.extraction_total`, and `timing.actor_total`. These logs are designed to help verify each step of the actor and identify bottlenecks in RSS fetching, URL decoding, browser extraction, fallback extraction, and total runtime. ### Notes and limitations - Google News RSS is used only for discovery; final extraction runs against publisher pages, not Google-hosted article pages. - `maxSites` applies to unique news sites per language/region pair, not to the total number of articles globally across all pairs. - `daysBack` controls the search lookback window, and the final actor sets `chunkDays = daysBack`, so each edition is queried with a single time chunk covering the full requested range. - The extracted text depends on publisher HTML structure and on whether content is accessible to browser automation or plain HTTP requests. - Some publishers may block automation, enforce paywalls, or deliver reduced content to automated clients. - The `languageAndRegion` choices in the input schema can be represented as a practical whitelist, but Google does not publish a stable, official, exhaustive list of all supported `ceid` combinations. ### Client code examples #### Node.js ```js import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with API token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "searchQuery": "\"samsung galaxy s25\" -ultra", "maxConcurrency": 10, "languageAndRegion": [ "IT:it", "US:en", "LT:lt" ], "maxSites": 50, "daysBack": 7, "decodeInterval": 1 }; (async () => { // Run the Actor and wait for it to finish const run = await client.actor("nQxTzKe5yrbyzysYh").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); })(); ``` #### Python ```python from apify_client import ApifyClient ## Initialize the ApifyClient with your API token client = ApifyClient("") ## Prepare the Actor input run_input = { "searchQuery": "\"samsung galaxy s25\" -ultra", "maxConcurrency": 10, "languageAndRegion": [ "IT:it", "US:en", "LT:lt", ], "maxSites": 50, "daysBack": 7, "decodeInterval": 1, } ## Run the Actor and wait for it to finish run = client.actor("nQxTzKe5yrbyzysYh").call(run_input=run_input) ## Fetch and print Actor results from the run's dataset (if there are any) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) ``` #### cURL ```bash ## Set API token API_TOKEN= ## Prepare Actor input cat > input.json <<'EOF' { "searchQuery": "\"samsung galaxy s25\" -ultra", "maxConcurrency": 10, "languageAndRegion": [ "IT:it", "US:en", "LT:lt" ], "maxSites": 50, "daysBack": 7, "decodeInterval": 1 } EOF ## Run the Actor curl "https://api.apify.com/v2/acts/nQxTzKe5yrbyzysYh/runs?token=$API_TOKEN" \ -X POST \ -d @input.json \ -H 'Content-Type: application/json' ``` # Actor input Schema ## `searchQuery` (type: `string`): Enter search query as you would write it to Google News search bar. You can use advanced operators such as intitle, inurl, site, exclude operator -, exact match with double-quotes, AND, OR and more. ## `maxConcurrency` (type: `integer`): Maximum number of pages opened simultaneously in the Playwright browser. This value represents the maximum number of browser pages or tabs processed in parallel, not the number of system threads. Max value is capped to 50. ## `languageAndRegion` (type: `array`): Practical whitelist of commonly used Google News country/language pairs in the format CC:ll. This is not an official or complete list of all combinations supported by Google News. ## `maxSites` (type: `integer`): Maximum number of unique news sites returned for each language/region pair. ## `daysBack` (type: `integer`): How many days back to search. chunkDays is always set equal to daysBack. ## `decodeInterval` (type: `integer`): Interval passed to googlenewsdecoder. Usually keep 1. ## Actor input object example ```json { "searchQuery": "Enter search query as you would write it to Google News search bar", "maxConcurrency": 10, "languageAndRegion": [ "IT:it", "US:en" ], "maxSites": 20, "daysBack": 7, "decodeInterval": 1 } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "searchQuery": "Enter search query as you would write it to Google News search bar", "maxConcurrency": 10, "languageAndRegion": [ "IT:it", "US:en" ], "maxSites": 20, "daysBack": 7, "decodeInterval": 1 }; // Run the Actor and wait for it to finish const run = await client.actor("ilborso/ai-news-content-crawler").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "searchQuery": "Enter search query as you would write it to Google News search bar", "maxConcurrency": 10, "languageAndRegion": [ "IT:it", "US:en", ], "maxSites": 20, "daysBack": 7, "decodeInterval": 1, } # Run the Actor and wait for it to finish run = client.actor("ilborso/ai-news-content-crawler").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "searchQuery": "Enter search query as you would write it to Google News search bar", "maxConcurrency": 10, "languageAndRegion": [ "IT:it", "US:en" ], "maxSites": 20, "daysBack": 7, "decodeInterval": 1 }' | apify call ilborso/ai-news-content-crawler --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=ilborso/ai-news-content-crawler", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "AI News Content Crawler", "description": "This Apify actor searches Google News RSS for one or more country/language editions, collects article URLs, opens each news page, extracts clean text, and stores one dataset item per processed URL. It uses Google News RSS discovery with automatic HTTP fallback when browser rendering fails.", "version": "0.0", "x-build-id": "CIEJGWLccZH2tbjn2" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/ilborso~ai-news-content-crawler/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-ilborso-ai-news-content-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/ilborso~ai-news-content-crawler/runs": { "post": { "operationId": "runs-sync-ilborso-ai-news-content-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/ilborso~ai-news-content-crawler/run-sync": { "post": { "operationId": "run-sync-ilborso-ai-news-content-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "searchQuery", "languageAndRegion" ], "properties": { "searchQuery": { "title": "Search query", "type": "string", "description": "Enter search query as you would write it to Google News search bar. You can use advanced operators such as intitle, inurl, site, exclude operator -, exact match with double-quotes, AND, OR and more." }, "maxConcurrency": { "title": "Max concurrency", "minimum": 1, "maximum": 50, "type": "integer", "description": "Maximum number of pages opened simultaneously in the Playwright browser. This value represents the maximum number of browser pages or tabs processed in parallel, not the number of system threads. Max value is capped to 50.", "default": 10 }, "languageAndRegion": { "title": "Language and region", "uniqueItems": true, "type": "array", "description": "Practical whitelist of commonly used Google News country/language pairs in the format CC:ll. This is not an official or complete list of all combinations supported by Google News.", "items": { "type": "string", "enum": [ "AR:es", "AT:de", "AU:en", "BE:fr", "BE:nl", "BG:bg", "BR:pt", "CA:en", "CA:fr", "CH:de", "CH:fr", "CH:it", "CL:es", "CO:es", "CZ:cs", "DE:de", "DK:da", "EE:et", "EG:ar", "ES:es", "FI:fi", "FR:fr", "GB:en", "GR:el", "HK:zh", "HR:hr", "HU:hu", "ID:id", "IE:en", "IL:he", "IN:en", "IN:hi", "IT:it", "JP:ja", "KR:ko", "LT:lt", "LV:lv", "MA:fr", "MX:es", "MY:en", "NG:en", "NL:nl", "NO:no", "NZ:en", "PE:es", "PH:en", "PK:en", "PL:pl", "PT:pt", "RO:ro", "RS:sr", "RU:ru", "SA:ar", "SE:sv", "SG:en", "SI:sl", "SK:sk", "TH:th", "TR:tr", "TW:zh", "UA:uk", "US:en", "VE:es", "VN:vi", "ZA:en" ] }, "default": [ "IT:it" ] }, "maxSites": { "title": "Max sites", "minimum": 1, "type": "integer", "description": "Maximum number of unique news sites returned for each language/region pair.", "default": 50 }, "daysBack": { "title": "Days back", "minimum": 1, "type": "integer", "description": "How many days back to search. chunkDays is always set equal to daysBack.", "default": 7 }, "decodeInterval": { "title": "Decode interval", "minimum": 1, "type": "integer", "description": "Interval passed to googlenewsdecoder. Usually keep 1.", "default": 1 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```