# News Aggregator & Media Monitor (`second_coming/news-aggregator-monitor`) Actor Scrape and aggregate articles from Google News, RSS feeds, and major news sites with keyword filtering, deduplication, and sentiment analysis. PR and media monitoring tool. - **URL**: https://apify.com/second\_coming/news-aggregator-monitor.md - **Developed by:** [Richard P](https://apify.com/second_coming) (community) - **Categories:** News, Marketing - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## News Aggregator & Media Monitor An Apify Actor for news aggregation and media monitoring. Scrapes and aggregates articles from **Google News**, **RSS/Atom feeds**, and **direct article URLs** with keyword filtering, deduplication, and sentiment analysis. ### Features - **Multi-source aggregation** — Search Google News, fetch RSS/Atom feeds, and extract content from direct article URLs in a single run - **Keyword filtering** — Search by one or more keywords across all sources - **Deduplication** — Automatically deduplicates articles by URL across sources - **Sentiment analysis** — Keyword-based positive / negative / neutral classification with confidence scores - **Full content extraction** — Optionally extract full article body text using JSON-LD, meta tags, and semantic HTML selectors - **Date filtering** — Only include articles published within a configurable number of days - **Rate limiting** — 1-second delay between requests to the same source to avoid being blocked - **Graceful abort handling** — Clean shutdown when the user stops the Actor on the Apify platform - **Structured output** — Clean JSON output with consistent schema across all sources ### Input The Actor accepts the following input fields: | Field | Type | Default | Description | |-------|------|---------|-------------| | `keywords` | `string[]` | `["technology"]` | Keywords to search for across all sources | | `sources` | `string[]` | `["google_news"]` | Sources to query: `google_news`, `rss`, `direct` | | `rssFeeds` | `string[]` | `[]` | RSS/Atom feed URLs (required when `rss` is a source) | | `directUrls` | `string[]` | `[]` | Direct article URLs (required when `direct` is a source) | | `maxArticles` | `integer` | `100` | Maximum number of articles to return | | `daysBack` | `integer` | `1` | Maximum age of articles in days | | `includeFullContent` | `boolean` | `false` | Whether to fetch full article body text | | `sentimentAnalysis` | `boolean` | `true` | Whether to perform sentiment classification | #### Example Input ```json { "keywords": ["artificial intelligence", "machine learning"], "sources": ["google_news", "rss"], "rssFeeds": [ "https://feeds.bbci.co.uk/news/rss.xml", "https://rss.nytimes.com/services/xml/rss/nyt/HomePage.xml" ], "maxArticles": 50, "daysBack": 3, "includeFullContent": false, "sentimentAnalysis": true } ```` ### Output Each article in the result dataset contains: | Field | Type | Description | |-------|------|-------------| | `title` | `string` | Article headline | | `url` | `string` | Canonical URL of the article | | `source` | `string` | Publication or news outlet name | | `summary` | `string` | Short description or snippet | | `fullContent` | `string` | Full article body text (only when `includeFullContent` is enabled) | | `author` | `string` | Article author or byline | | `publishDate` | `string` | ISO 8601 publication date | | `sourceType` | `string` | How the article was found: `google_news`, `rss`, or `direct` | | `keywords` | `string[]` | Keywords that matched this article | | `sentiment` | `object` | `{ "label": "positive"\|"negative"\|"neutral", "confidence": 0.0–1.0 }` | | `imageUrl` | `string` | URL to the lead/hero image | | `categories` | `string[]` | Article categories or tags (RSS/Atom feeds) | #### Example Output ```json { "title": "AI Breakthrough: New Model Achieves Human-Level Reasoning", "url": "https://example.com/ai-breakthrough", "source": "Tech News Daily", "summary": "Researchers have developed a new AI model that can reason at human levels...", "author": "Jane Smith", "publishDate": "2026-07-04T10:30:00+00:00", "sourceType": "google_news", "keywords": ["artificial intelligence"], "sentiment": { "label": "positive", "confidence": 0.875 }, "imageUrl": "https://example.com/images/ai-model.jpg", "categories": ["Technology", "Artificial Intelligence"] } ``` ### Sentiment Analysis Methodology The sentiment classifier uses a **keyword-frequency approach**: 1. **Positive keywords** (~100 terms): growth, success, breakthrough, profit, innovation, milestone, recovery, promising, etc. 2. **Negative keywords** (~120 terms): decline, loss, failure, crisis, risk, crash, bankruptcy, scandal, etc. 3. The title and summary text is tokenised and each token is matched against the keyword lists. 4. **Classification rule:** - More positive hits → `positive`; confidence = positive / (positive + negative) - More negative hits → `negative`; confidence = negative / (positive + negative) - Equal or zero hits → `neutral`; confidence = 0.0 5. A small dampening factor is applied when fewer than 3 total hits are found (penalties low-confidence classifications from sparse text). **Limitations:** This is a simple bag-of-words approach. It does not handle negation ("not good" → negative), sarcasm, or context-dependent sentiment. It is designed for speed and reliability rather than state-of-the-art accuracy. ### Source Details #### Google News - Queries `news.google.com/search?q=KEYWORD&hl=en&gl=US` - Parses `
` elements using multiple CSS selector fallbacks - Resolves Google News redirect URLs to their canonical article URLs - Extracts: title, URL, source name, summary snippet, relative date, image #### RSS/Atom Feeds - Supports both RSS 2.0 (`` elements) and Atom (`` elements) - Handles RSS enclosures and `media:content` for images - Extracts: title, URL, description, author, pubDate, categories, image #### Direct Article Extraction - Extracts full article content from a single URL - **JSON-LD:** Parses `application/ld+json` for structured article data (NewsArticle, Article, BlogPosting) - **Meta tags:** Reads Open Graph (`og:title`, `og:description`, `og:image`), Twitter Card, and standard meta tags - **Body extraction:** Uses semantic HTML selectors: `
`, `div[role="main"]`, `.article-body`, `.entry-content`, `.story-body`, and a `

`-based fallback ### Local Development #### Prerequisites - Python 3.13+ - Apify CLI installed globally ```bash npm install -g apify-cli ``` #### Setup ```bash ## Create a virtual environment python3 -m venv venv source venv/bin/activate ## Install dependencies pip install -r requirements.txt ``` #### Run Locally ```bash ## From the actor directory apify run --purge ``` To customise the input, edit `.actor/input_schema.json` default values or create a `INPUT.json` in the `storage/` directory. #### Project Structure ``` news-aggregator-monitor/ ├── .actor/ │ ├── actor.json # Actor manifest │ ├── input_schema.json # Input schema │ ├── output_schema.json # Output schema │ └── dataset_schema.json # Dataset schema with views ├── my_actor/ │ ├── __init__.py # Package init │ ├── __main__.py # Entry point (asyncio.run(main())) │ ├── main.py # Actor orchestrator │ ├── models.py # Data models (Article, SentimentResult) │ ├── sentiment.py # Keyword-based sentiment analysis │ └── sources.py # Google News, RSS, direct article scrapers ├── Dockerfile # Container image ├── requirements.txt # Python dependencies ├── README.md # This file ├── AGENTS.md # Apify Actor development guide ├── storage/ # Local run data ├── .dockerignore └── .gitignore ``` ### Deployment ```bash ## Push to Apify Platform apify push ``` ### Use Cases - **PR & Media Monitoring** — Track mentions of your brand, products, or executives across news sources - **Competitor Intelligence** — Monitor keywords related to competitors' products, launches, and financial performance - **Market Research** — Aggregate news on specific industries, technologies, or trends - **Academic Research** — Collect news article metadata for content analysis and media studies - **Investment Research** — Monitor financial news for sentiment trends on stocks, sectors, or commodities # Actor input Schema ## `keywords` (type: `array`): Keywords to search for in news sources. Each keyword is treated as a separate query. ## `sources` (type: `array`): Which news sources to query: google\_news, rss, direct. ## `rssFeeds` (type: `array`): RSS or Atom feed URLs to fetch articles from. ## `directUrls` (type: `array`): Specific article URLs to extract full content from. ## `maxArticles` (type: `integer`): Maximum number of articles to return. ## `daysBack` (type: `integer`): Maximum age of articles in days. ## `includeFullContent` (type: `boolean`): Fetch full article body text (slower). ## `sentimentAnalysis` (type: `boolean`): Perform keyword-based sentiment classification. ## `proxyConfiguration` (type: `object`): Apify proxy configuration for bypassing geo-restrictions. ## Actor input object example ```json { "keywords": [ "artificial intelligence", "stock market" ], "sources": [ "google_news", "rss" ], "rssFeeds": [ "https://feeds.bbci.co.uk/news/rss.xml", "https://rss.nytimes.com/services/xml/rss/nyt/HomePage.xml" ], "directUrls": [], "maxArticles": 100, "daysBack": 1, "includeFullContent": false, "sentimentAnalysis": true, "proxyConfiguration": { "useApifyProxy": false } } ``` # Actor output Schema ## `results` (type: `string`): Link to the dataset containing aggregated articles # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "keywords": [ "artificial intelligence", "stock market" ], "sources": [ "google_news", "rss" ], "rssFeeds": [ "https://feeds.bbci.co.uk/news/rss.xml", "https://rss.nytimes.com/services/xml/rss/nyt/HomePage.xml" ], "proxyConfiguration": { "useApifyProxy": false } }; // Run the Actor and wait for it to finish const run = await client.actor("second_coming/news-aggregator-monitor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "keywords": [ "artificial intelligence", "stock market", ], "sources": [ "google_news", "rss", ], "rssFeeds": [ "https://feeds.bbci.co.uk/news/rss.xml", "https://rss.nytimes.com/services/xml/rss/nyt/HomePage.xml", ], "proxyConfiguration": { "useApifyProxy": False }, } # Run the Actor and wait for it to finish run = client.actor("second_coming/news-aggregator-monitor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "keywords": [ "artificial intelligence", "stock market" ], "sources": [ "google_news", "rss" ], "rssFeeds": [ "https://feeds.bbci.co.uk/news/rss.xml", "https://rss.nytimes.com/services/xml/rss/nyt/HomePage.xml" ], "proxyConfiguration": { "useApifyProxy": false } }' | apify call second_coming/news-aggregator-monitor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=second_coming/news-aggregator-monitor", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "News Aggregator & Media Monitor", "description": "Scrape and aggregate articles from Google News, RSS feeds, and major news sites with keyword filtering, deduplication, and sentiment analysis. PR and media monitoring tool.", "version": "0.0", "x-build-id": "BDGcSBBfa3GPN5RDM" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/second_coming~news-aggregator-monitor/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-second_coming-news-aggregator-monitor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/second_coming~news-aggregator-monitor/runs": { "post": { "operationId": "runs-sync-second_coming-news-aggregator-monitor", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/second_coming~news-aggregator-monitor/run-sync": { "post": { "operationId": "run-sync-second_coming-news-aggregator-monitor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "keywords": { "title": "Keywords", "uniqueItems": true, "type": "array", "description": "Keywords to search for in news sources. Each keyword is treated as a separate query.", "items": { "type": "string" }, "default": [] }, "sources": { "title": "News Sources", "uniqueItems": true, "type": "array", "description": "Which news sources to query: google_news, rss, direct.", "items": { "type": "string" }, "default": [ "google_news" ] }, "rssFeeds": { "title": "RSS/Atom Feed URLs", "type": "array", "description": "RSS or Atom feed URLs to fetch articles from.", "items": { "type": "string" }, "default": [] }, "directUrls": { "title": "Direct Article URLs", "type": "array", "description": "Specific article URLs to extract full content from.", "items": { "type": "string" }, "default": [] }, "maxArticles": { "title": "Max Articles", "minimum": 1, "maximum": 1000, "type": "integer", "description": "Maximum number of articles to return.", "default": 100 }, "daysBack": { "title": "Days Back", "minimum": 1, "maximum": 30, "type": "integer", "description": "Maximum age of articles in days.", "default": 1 }, "includeFullContent": { "title": "Include Full Content", "type": "boolean", "description": "Fetch full article body text (slower).", "default": false }, "sentimentAnalysis": { "title": "Sentiment Analysis", "type": "boolean", "description": "Perform keyword-based sentiment classification.", "default": true }, "proxyConfiguration": { "title": "Proxy Configuration", "type": "object", "description": "Apify proxy configuration for bypassing geo-restrictions." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```