RSS Feed Scraper & Keyword Monitor

Turn any list of RSS/Atom feeds — or just website homepages — into a clean, deduplicated, keyword-filtered stream of news items with one normalized schema.

Built for: brand monitoring, competitor tracking, niche news aggregation, content pipelines, and feeding fresh articles to LLM/RAG systems.

Why this monitor

News scrapers on the marketplace are mostly $20+/month rentals that wrap a single source and break when it changes. Feeds don't break — they are designed for programmatic consumption. This actor makes them effortless:

• Feed auto-discovery. Don't know the feed URL? Give the homepage; feeds are found via <link rel="alternate"> tags, with fallback probing of common paths (/feed, /rss.xml, /atom.xml...).
• One schema for RSS 2.0 and Atom. Title, link, GUID, ISO-8601 published date, author, categories, summary and full content as plain text — no XML wrangling on your side.
• Keyword filtering server-side. Only pay for items that mention your keywords; matches are recorded per item.
• Cross-feed deduplication. Overlapping feeds (aggregators, category feeds) don't produce duplicate items.
• Pay per item, no rental. Run it hourly on a schedule for pennies.

Input

Field	Type	Default	Notes
feedUrls	array	[]	Direct RSS/Atom URLs.
siteUrls	array	[]	Homepages — feeds auto-discovered.
keywords	array	[]	Keep only items matching ≥1 keyword (case-insensitive). Empty = keep all.
maxItemsPerFeed	integer	50	Cap per feed.
includeContent	boolean	true	Full content text when the feed provides it.
deduplicate	boolean	true	Drop repeated links/GUIDs within the run.

{
    "feedUrls": ["https://techcrunch.com/feed/"],
    "siteUrls": ["https://www.theverge.com"],
    "keywords": ["insurance", "climate"],
    "maxItemsPerFeed": 50
}

Output schema

One dataset item per feed entry:

{
    "feedUrl": "https://techcrunch.com/feed/",
    "feedTitle": "TechCrunch",
    "siteUrl": null,
    "title": "Insurtech startup raises $40M Series B",
    "url": "https://techcrunch.com/2026/07/03/insurtech-startup...",
    "guid": "https://techcrunch.com/?p=2751...",
    "publishedAt": "2026-07-03T14:02:11.000Z",
    "author": "Jane Reporter",
    "categories": ["Fintech", "Insurance"],
    "summaryText": "The round was led by...",
    "contentText": "Full article text when the feed publishes it...",
    "matchedKeywords": ["insurance"],
    "fetchedAt": "2026-07-04T10:00:00.000Z"
}

Pricing

Pay per result. 1 result = 1 feed item kept (after keyword filtering and deduplication — filtered-out items cost nothing).

Items	Cost at $0.50 / 1,000
1,000	$0.50
20,000 (50 feeds, hourly, a day)	$10.00

Compare: single-source news actors at $20/month rental before usage.

Run summary

Every run writes a RUN_SUMMARY record to the key-value store: per-feed item counts and errors, per-site discovery counts and robotsBlocked status, total items pushed, and feed failure count. No silent zero-result runs — if a feed 404s, times out, or is blocked by robots.txt, it shows up here with the reason.

Scheduling

Pair with an Apify Schedule (e.g. every 30 min) and a webhook/integration (Slack, Google Sheets, Make, Zapier) for a complete monitoring pipeline. deduplicate works within a run; for cross-run dedup, key on url in your destination.

Compliance & fair use

RSS/Atom feeds exist to be fetched programmatically. One polite request per feed per run, standard retries only. No paywall circumvention — the actor returns what the publisher chose to publish in the feed. For siteUrls, the site's robots.txt is checked first — the homepage fetch and every auto-discovery probe path (/feed, /rss.xml, etc.) is skipped if disallowed, and this is recorded per site in the run summary.

Using from API / MCP / AI agents

curl -X POST "https://api.apify.com/v2/acts/<ACTOR_ID>/run-sync-get-dataset-items?token=<TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{ "feedUrls": ["https://hnrss.org/frontpage"], "keywords": ["postgres"] }'

Works with the Apify MCP server — agents can ask "what's new about X across these sources" in one tool call.

FAQ

Full article text or just the feed content? What the feed publishes. Many feeds carry full content:encoded; truncated feeds carry summaries. Fetching and parsing article pages beyond the feed is a different (heavier) job — pipe url into the Website to Markdown actor if you need it.

Google News? Its robots.txt disallows automated fetching of its RSS endpoints, so this actor does not special-case it. Use publisher feeds directly — they are richer anyway.

Changelog

• 0.1.0 (2026-07) — Initial release: RSS 2.0 + Atom, feed auto-discovery with path probing, keyword filter with per-item match records, cross-feed dedup, plain-text content extraction, robots.txt-gated homepage discovery, per-run RUN_SUMMARY.

---

Maintained by Sturdy Data — boring, reliable scrapers. Report issues in the Issues tab; we respond within 24 hours.