News Aggregator & Media Monitor

An Apify Actor for news aggregation and media monitoring. Scrapes and aggregates articles from Google News, RSS/Atom feeds, and direct article URLs with keyword filtering, deduplication, and sentiment analysis.

Features

• Multi-source aggregation — Search Google News, fetch RSS/Atom feeds, and extract content from direct article URLs in a single run
• Keyword filtering — Search by one or more keywords across all sources
• Deduplication — Automatically deduplicates articles by URL across sources
• Sentiment analysis — Keyword-based positive / negative / neutral classification with confidence scores
• Full content extraction — Optionally extract full article body text using JSON-LD, meta tags, and semantic HTML selectors
• Date filtering — Only include articles published within a configurable number of days
• Rate limiting — 1-second delay between requests to the same source to avoid being blocked
• Graceful abort handling — Clean shutdown when the user stops the Actor on the Apify platform
• Structured output — Clean JSON output with consistent schema across all sources

Input

The Actor accepts the following input fields:

Field	Type	Default	Description
keywords	string[]	["technology"]	Keywords to search for across all sources
sources	string[]	["google_news"]	Sources to query: google_news, rss, direct
rssFeeds	string[]	[]	RSS/Atom feed URLs (required when rss is a source)
directUrls	string[]	[]	Direct article URLs (required when direct is a source)
maxArticles	integer	100	Maximum number of articles to return
daysBack	integer	1	Maximum age of articles in days
includeFullContent	boolean	false	Whether to fetch full article body text
sentimentAnalysis	boolean	true	Whether to perform sentiment classification

Example Input

{
  "keywords": ["artificial intelligence", "machine learning"],
  "sources": ["google_news", "rss"],
  "rssFeeds": [
    "https://feeds.bbci.co.uk/news/rss.xml",
    "https://rss.nytimes.com/services/xml/rss/nyt/HomePage.xml"
  ],
  "maxArticles": 50,
  "daysBack": 3,
  "includeFullContent": false,
  "sentimentAnalysis": true
}

Output

Each article in the result dataset contains:

Field	Type	Description
title	string	Article headline
url	string	Canonical URL of the article
source	string	Publication or news outlet name
summary	string	Short description or snippet
fullContent	string	Full article body text (only when includeFullContent is enabled)
author	string	Article author or byline
publishDate	string	ISO 8601 publication date
sourceType	string	How the article was found: google_news, rss, or direct
keywords	string[]	Keywords that matched this article
sentiment	object	{ "label": "positive"|"negative"|"neutral", "confidence": 0.0–1.0 }
imageUrl	string	URL to the lead/hero image
categories	string[]	Article categories or tags (RSS/Atom feeds)

Example Output

{
  "title": "AI Breakthrough: New Model Achieves Human-Level Reasoning",
  "url": "https://example.com/ai-breakthrough",
  "source": "Tech News Daily",
  "summary": "Researchers have developed a new AI model that can reason at human levels...",
  "author": "Jane Smith",
  "publishDate": "2026-07-04T10:30:00+00:00",
  "sourceType": "google_news",
  "keywords": ["artificial intelligence"],
  "sentiment": {
    "label": "positive",
    "confidence": 0.875
  },
  "imageUrl": "https://example.com/images/ai-model.jpg",
  "categories": ["Technology", "Artificial Intelligence"]
}

Sentiment Analysis Methodology

The sentiment classifier uses a keyword-frequency approach:

1. Positive keywords (~100 terms): growth, success, breakthrough, profit, innovation, milestone, recovery, promising, etc.
2. Negative keywords (~120 terms): decline, loss, failure, crisis, risk, crash, bankruptcy, scandal, etc.
3. The title and summary text is tokenised and each token is matched against the keyword lists.
4. Classification rule:
   • More positive hits → positive; confidence = positive / (positive + negative)
   • More negative hits → negative; confidence = negative / (positive + negative)
   • Equal or zero hits → neutral; confidence = 0.0
5. A small dampening factor is applied when fewer than 3 total hits are found (penalties low-confidence classifications from sparse text).

Limitations: This is a simple bag-of-words approach. It does not handle negation ("not good" → negative), sarcasm, or context-dependent sentiment. It is designed for speed and reliability rather than state-of-the-art accuracy.

Source Details

Google News

• Queries news.google.com/search?q=KEYWORD&hl=en&gl=US
• Parses <article> elements using multiple CSS selector fallbacks
• Resolves Google News redirect URLs to their canonical article URLs
• Extracts: title, URL, source name, summary snippet, relative date, image

RSS/Atom Feeds

• Supports both RSS 2.0 (<item> elements) and Atom (<entry> elements)
• Handles RSS enclosures and media:content for images
• Extracts: title, URL, description, author, pubDate, categories, image

Direct Article Extraction

• Extracts full article content from a single URL
• JSON-LD: Parses application/ld+json for structured article data (NewsArticle, Article, BlogPosting)
• Meta tags: Reads Open Graph (og:title, og:description, og:image), Twitter Card, and standard meta tags
• Body extraction: Uses semantic HTML selectors: <article>, div[role="main"], .article-body, .entry-content, .story-body, and a <p>-based fallback

Local Development

Prerequisites

• Python 3.13+
• Apify CLI installed globally

$npm install -g apify-cli

Setup

# Create a virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

Run Locally

# From the actor directory
apify run --purge

To customise the input, edit .actor/input_schema.json default values or create a INPUT.json in the storage/ directory.

Project Structure

news-aggregator-monitor/
├── .actor/
│   ├── actor.json              # Actor manifest
│   ├── input_schema.json       # Input schema
│   ├── output_schema.json      # Output schema
│   └── dataset_schema.json     # Dataset schema with views
├── my_actor/
│   ├── __init__.py             # Package init
│   ├── __main__.py             # Entry point (asyncio.run(main()))
│   ├── main.py                 # Actor orchestrator
│   ├── models.py               # Data models (Article, SentimentResult)
│   ├── sentiment.py            # Keyword-based sentiment analysis
│   └── sources.py              # Google News, RSS, direct article scrapers
├── Dockerfile                  # Container image
├── requirements.txt            # Python dependencies
├── README.md                   # This file
├── AGENTS.md                   # Apify Actor development guide
├── storage/                    # Local run data
├── .dockerignore
└── .gitignore

Deployment

# Push to Apify Platform
apify push

Use Cases

• PR & Media Monitoring — Track mentions of your brand, products, or executives across news sources
• Competitor Intelligence — Monitor keywords related to competitors' products, launches, and financial performance
• Market Research — Aggregate news on specific industries, technologies, or trends
• Academic Research — Collect news article metadata for content analysis and media studies
• Investment Research — Monitor financial news for sentiment trends on stocks, sectors, or commodities