# Multi-Source News & Content Scraper (`moving_beacon-owner1/rss-feeds----multi-source-news-content-scraper`) Actor Multi-Source News & Content Scraper. Aggregates articles from multiple RSS/Atom feeds simultaneously. Includes 60+ pre-built news source presets and supports custom feed URLs. No API key required. - **URL**: https://apify.com/moving\_beacon-owner1/rss-feeds----multi-source-news-content-scraper.md - **Developed by:** [Jamshaid Arif](https://apify.com/moving_beacon-owner1) (community) - **Categories:** News, Automation, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing from $10.00 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## πŸ“° RSS Feeds β€” Multi-Source News & Content Scraper β€” Apify Actor Aggregates articles from **multiple RSS/Atom feeds** simultaneously. Includes **60+ pre-built news source presets** and supports custom feed URLs. **No API key required.** ### Features - **60+ preset feeds** β€” TechCrunch, BBC, Reuters, Wired, NYT, The Guardian, Hacker News, arXiv, NASA, ESPN, Reddit, and many more. - **Custom feed URLs** β€” add any RSS/Atom feed alongside presets. - **Keyword filtering** β€” include articles matching specific keywords, or exclude unwanted terms. - **Date filtering** β€” only articles from the last N hours, or published after a specific date. - **Category/tag filtering** β€” filter by RSS categories or tags. - **Deduplication** β€” removes duplicate articles appearing across multiple feeds. - **3 output formats** β€” enriched, minimal, raw. - **Auto-retry** β€” each feed retried up to 2 times on failure. - **Per-feed stats** β€” summary shows fetched/kept counts and status per source. ### Preset Categories | Category | Sources | |---|---| | **Tech News** | TechCrunch, Wired, Ars Technica, The Verge, Mashable, Engadget, ZDNet, VentureBeat, MIT Tech Review | | **World News** | BBC World, Reuters, Al Jazeera, NPR, CNN, NYT, The Guardian, WSJ, The Economist | | **Developer** | Hacker News, Lobsters, DEV.to, HackerNoon, CSS-Tricks, Smashing Magazine, freeCodeCamp | | **Company Blogs** | GitHub, AWS, Google AI, OpenAI, Anthropic | | **Languages** | Python Insider, Go Blog, Rust Blog | | **Science** | Nature, Science Daily, Phys.org, NASA, Space.com, arXiv CS/AI | | **Sports** | ESPN, BBC Sport | | **Business** | Forbes, Bloomberg Markets, CoinDesk, The Block | | **Social** | Reddit r/technology, r/programming, r/worldnews, Product Hunt, Medium | ### Input Examples #### AI News Aggregator ```json { "presetFeeds": ["techcrunch", "wired", "mit_tech_review", "openai_blog", "anthropic_blog", "google_ai_blog", "arxiv_cs_ai"], "keywordFilter": "AI, artificial intelligence, LLM, GPT, machine learning", "maxAgeHours": 72, "sortBy": "date_desc", "outputFormat": "enriched" } ```` #### Developer News Feed ```json { "presetFeeds": ["hacker_news", "lobsters", "dev_to", "github_blog", "python_insider", "rust_blog"], "excludeKeywords": "sponsored, hiring", "maxArticlesPerFeed": 20, "sortBy": "date_desc" } ``` #### World News Monitor ```json { "presetFeeds": ["bbc_world", "reuters_world", "al_jazeera", "nyt_world", "guardian_world"], "maxAgeHours": 24, "deduplicate": true, "sortBy": "date_desc" } ``` #### Custom Feeds + Presets ```json { "presetFeeds": ["techcrunch"], "customFeedUrls": "https://example.com/feed.xml\nhttps://myblog.com/rss", "outputFormat": "enriched" } ``` ### Enriched Output Fields | Field | Example | |---|---| | `id` | `1` | | `title` | `OpenAI Announces GPT-5 with Reasoning` | | `link` | `https://techcrunch.com/2025/…` | | `domain` | `techcrunch.com` | | `source` | `TechCrunch` | | `source_category` | `Tech` | | `author` | `Jane Smith` | | `published` | `2025-04-05T10:30:00+00:00` | | `published_date` | `2025-04-05` | | `published_time` | `10:30:00` | | `summary` | `OpenAI has unveiled its latest…` | | `content_preview` | `The full article text preview…` | | `categories` | `AI, Machine Learning, OpenAI` | | `image_url` | `https://…/header.jpg` | | `word_count` | `342` | | `guid` | `https://techcrunch.com/?p=12345` | | `feed_url` | `https://techcrunch.com/feed/` | #### Minimal Format `title`, `link`, `source`, `category`, `published` #### Raw Format Full feedparser entry structure with `_source`, `_source_category`, `_feed_url` added. ### Filtering #### Keywords (Include) Comma-separated, case-insensitive. Articles kept if title OR summary contains **ANY** keyword. `"AI, machine learning, GPT"` β†’ keeps articles mentioning any of these. #### Keywords (Exclude) Articles removed if title OR summary contains **ANY** excluded keyword. `"sponsored, advertisement, hiring"` β†’ removes promotional content. #### Date Filtering - **Max Age Hours**: only articles from the last N hours (e.g. `24` = last day) - **Published After**: only articles after a date (`2025-01-01`) #### Category Filter Comma-separated RSS tags/categories. Articles kept if tagged with **ANY** matching category. ### Run Summary Saved to key-value store as `summary`: ```json { "total_feeds": 7, "total_articles": 142, "feeds_status": { "TechCrunch": {"fetched": 30, "kept": 12, "status": "ok"}, "Wired": {"fetched": 25, "kept": 8, "status": "ok"}, "OpenAI Blog": {"fetched": 0, "kept": 0, "status": "failed"} }, "source_breakdown": {"TechCrunch": 12, "Wired": 8, ...}, "category_breakdown": {"Tech": 45, "AI": 28, ...} } ``` # Actor input Schema ## `presetFeeds` (type: `array`): Select from popular pre-configured RSS feeds. ## `customFeedUrls` (type: `string`): Add your own RSS/Atom feed URLs, one per line. These are fetched in addition to any preset feeds selected above. ## `keywordFilter` (type: `string`): Only keep articles whose title or summary contains ANY of these keywords (comma-separated, case-insensitive). E.g. 'AI, machine learning, GPT'. Leave empty for no filter. ## `excludeKeywords` (type: `string`): Remove articles whose title or summary contains ANY of these keywords (comma-separated). E.g. 'sponsored, advertisement'. ## `maxAgeHours` (type: `integer`): Only keep articles published within the last N hours. 0 = no age filter. ## `publishedAfter` (type: `string`): Only articles published after this date (YYYY-MM-DD). Overrides max age filter if set. ## `categoryFilter` (type: `string`): Only keep articles tagged with ANY of these categories (comma-separated, case-insensitive). E.g. 'technology, science, AI'. Leave empty for all. ## `deduplicate` (type: `boolean`): Remove duplicate articles (same title or same URL across feeds). ## `sortBy` (type: `string`): How to sort the output articles. ## `outputFormat` (type: `string`): How to structure each article record. ## `maxArticlesPerFeed` (type: `integer`): Limit articles per individual feed. 0 = all available in the feed. ## `maxTotalArticles` (type: `integer`): Limit total output across all feeds. 0 = no limit. ## `addSequentialId` (type: `boolean`): Add auto-incrementing ID to each record. ## Actor input object example ```json { "presetFeeds": [ "techcrunch", "wired", "ars_technica" ], "maxAgeHours": 0, "deduplicate": true, "sortBy": "date_desc", "outputFormat": "enriched", "maxArticlesPerFeed": 0, "maxTotalArticles": 0, "addSequentialId": true } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("moving_beacon-owner1/rss-feeds----multi-source-news-content-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("moving_beacon-owner1/rss-feeds----multi-source-news-content-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call moving_beacon-owner1/rss-feeds----multi-source-news-content-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=moving_beacon-owner1/rss-feeds----multi-source-news-content-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Multi-Source News & Content Scraper", "description": "Multi-Source News & Content Scraper. Aggregates articles from multiple RSS/Atom feeds simultaneously. Includes 60+ pre-built news source presets and supports custom feed URLs. No API key required.", "version": "0.0", "x-build-id": "eirIOaS09A1RL5Zkc" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/moving_beacon-owner1~rss-feeds----multi-source-news-content-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-moving_beacon-owner1-rss-feeds----multi-source-news-content-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/moving_beacon-owner1~rss-feeds----multi-source-news-content-scraper/runs": { "post": { "operationId": "runs-sync-moving_beacon-owner1-rss-feeds----multi-source-news-content-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/moving_beacon-owner1~rss-feeds----multi-source-news-content-scraper/run-sync": { "post": { "operationId": "run-sync-moving_beacon-owner1-rss-feeds----multi-source-news-content-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "presetFeeds": { "title": "Preset News Sources", "type": "array", "description": "Select from popular pre-configured RSS feeds.", "items": { "type": "string", "enum": [ "techcrunch", "wired", "ars_technica", "the_verge", "mashable", "engadget", "zdnet", "venturebeat", "mit_tech_review", "bbc_world", "bbc_tech", "bbc_science", "bbc_business", "reuters_world", "reuters_tech", "reuters_business", "reuters_science", "al_jazeera", "npr_news", "npr_tech", "cnn_top", "nyt_home", "nyt_tech", "nyt_science", "nyt_world", "guardian_world", "guardian_tech", "guardian_science", "wsj_world", "economist", "hacker_news", "lobsters", "dev_to", "hackernoon", "css_tricks", "smashing_magazine", "freecodecamp", "github_blog", "aws_blog", "google_ai_blog", "openai_blog", "anthropic_blog", "python_insider", "golang_blog", "rust_blog", "nature", "science_daily", "phys_org", "nasa_breaking", "space_com", "arxiv_cs_ai", "espn", "bbc_sport", "forbes", "bloomberg_markets", "coindesk", "the_block_crypto", "product_hunt", "lifehacker", "medium_tech", "reddit_technology", "reddit_programming", "reddit_worldnews" ], "enumTitles": [ "TechCrunch", "Wired", "Ars Technica", "The Verge", "Mashable", "Engadget", "ZDNet", "VentureBeat", "MIT Technology Review", "BBC β€” World News", "BBC β€” Technology", "BBC β€” Science & Environment", "BBC β€” Business", "Reuters β€” World", "Reuters β€” Technology", "Reuters β€” Business", "Reuters β€” Science", "Al Jazeera", "NPR β€” News", "NPR β€” Technology", "CNN β€” Top Stories", "NYT β€” Home Page", "NYT β€” Technology", "NYT β€” Science", "NYT β€” World", "The Guardian β€” World", "The Guardian β€” Technology", "The Guardian β€” Science", "WSJ β€” World News", "The Economist", "Hacker News (hnrss)", "Lobsters", "DEV Community", "HackerNoon", "CSS-Tricks", "Smashing Magazine", "freeCodeCamp", "GitHub Blog", "AWS Blog", "Google AI Blog", "OpenAI Blog", "Anthropic Blog", "Python Insider", "Go Blog", "Rust Blog", "Nature β€” Latest Research", "Science Daily", "Phys.org", "NASA β€” Breaking News", "Space.com", "arXiv β€” CS / AI", "ESPN", "BBC β€” Sport", "Forbes", "Bloomberg β€” Markets", "CoinDesk β€” Crypto", "The Block β€” Crypto", "Product Hunt", "Lifehacker", "Medium β€” Technology", "Reddit β€” r/technology", "Reddit β€” r/programming", "Reddit β€” r/worldnews" ] }, "default": [ "techcrunch", "wired", "ars_technica" ] }, "customFeedUrls": { "title": "Custom Feed URLs", "type": "string", "description": "Add your own RSS/Atom feed URLs, one per line. These are fetched in addition to any preset feeds selected above." }, "keywordFilter": { "title": "Keyword Filter (Include)", "type": "string", "description": "Only keep articles whose title or summary contains ANY of these keywords (comma-separated, case-insensitive). E.g. 'AI, machine learning, GPT'. Leave empty for no filter." }, "excludeKeywords": { "title": "Keyword Filter (Exclude)", "type": "string", "description": "Remove articles whose title or summary contains ANY of these keywords (comma-separated). E.g. 'sponsored, advertisement'." }, "maxAgeHours": { "title": "Maximum Article Age (hours)", "minimum": 0, "type": "integer", "description": "Only keep articles published within the last N hours. 0 = no age filter.", "default": 0 }, "publishedAfter": { "title": "Published After", "type": "string", "description": "Only articles published after this date (YYYY-MM-DD). Overrides max age filter if set." }, "categoryFilter": { "title": "Category / Tag Filter", "type": "string", "description": "Only keep articles tagged with ANY of these categories (comma-separated, case-insensitive). E.g. 'technology, science, AI'. Leave empty for all." }, "deduplicate": { "title": "Deduplicate Articles", "type": "boolean", "description": "Remove duplicate articles (same title or same URL across feeds).", "default": true }, "sortBy": { "title": "Sort By", "enum": [ "date_desc", "date_asc", "source_asc", "title_asc" ], "type": "string", "description": "How to sort the output articles.", "default": "date_desc" }, "outputFormat": { "title": "Output Format", "enum": [ "enriched", "minimal", "raw" ], "type": "string", "description": "How to structure each article record.", "default": "enriched" }, "maxArticlesPerFeed": { "title": "Max Articles Per Feed", "minimum": 0, "type": "integer", "description": "Limit articles per individual feed. 0 = all available in the feed.", "default": 0 }, "maxTotalArticles": { "title": "Max Total Articles", "minimum": 0, "type": "integer", "description": "Limit total output across all feeds. 0 = no limit.", "default": 0 }, "addSequentialId": { "title": "Add Sequential ID", "type": "boolean", "description": "Add auto-incrementing ID to each record.", "default": true } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```