RSS & Atom Feed Scraper: Universal Content Monitor

RSS & Atom Feed Scraper reads any RSS, Atom or JSON feed on the web and returns one flat, identically-shaped record per item, title, link, author, published date, categories, media enclosures and the full article body.

Syndication feeds are the oldest public API on the internet: publishers maintain them on purpose, for machines. This Actor reads them directly. No proxy setup, no browser, no credentials to manage. Point it at a feed URL, or at a plain site homepage, the page's advertised feed is discovered and followed for you.

Agent skill: SKILL.md

https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/rss-feed-scraper.md

What you get

Output field	Meaning
requestedUrl, feedUrl	What you asked for, and the feed that was actually read after discovery and redirects
feedTitle, feedType	Channel title, and which format it turned out to be, rss, rss-1.0, atom or json
itemId	The feed's own identity for the item, RSS guid, Atom id, JSON Feed id
title, link	Headline and permalink, resolved to an absolute URL
author	dc:creator, Atom <author><name> or JSON Feed authors[], whichever the feed uses
publishedAt, updatedAt	Always ISO-8601, whether the feed wrote RFC-822 or ISO-8601
summary	Short description as plain text
contentText	Full article body as plain text, HTML stripped, entities decoded
contentHtml	The publisher's original markup, when you ask to keep it
categories	Tags and categories the item is filed under
enclosures	Attached media as { url, type, length }, podcast audio, video, PDFs
imageUrl	Lead image from media:content, media:thumbnail, itunes:image, an image enclosure, or the first <img> in the body
scrapedAt	Run timestamp

A RUN_SUMMARY record in the key-value store holds per-run counts, a per-feed breakdown, the filters used, and any source that failed.

Common use cases

• Competitor and newsroom monitoring. Watch dozens of blogs on a schedule and diff on itemId.
• News aggregation. One normalised schema across hundreds of publishers, so downstream code has one shape to handle.
• Content curation pipelines. Filter by keyword at ingest and hand the rest to a summariser.
• Brand monitoring. Track mentions across the feeds of trade press and forums.
• Archiving. Snapshot full article bodies before publishers rewrite or unpublish them.

Quick start

Three feeds, three different formats, everything they carry:

{
 "feedUrls": [
 "https://news.ycombinator.com/rss",
 "https://blog.rust-lang.org/feed.xml",
 "https://daringfireball.net/feeds/json"
 ]
}

Homepages instead of feeds, the Actor finds them:

{
 "feedUrls": ["https://arstechnica.com", "github.blog", "https://blog.rust-lang.org/"],
 "maxItemsPerFeed": 20
}

A filtered, recent, headline-only sweep for change detection:

{
 "feedUrls": ["https://feeds.bbci.co.uk/news/rss.xml"],
 "sinceDate": "2026-08-01",
 "keywordFilter": ["election", "budget"],
 "includeContent": false
}

Input

Field	Type	Default	Notes
feedUrls	array	-	Required. Feed URLs or site homepages. A bare host (github.blog) is accepted and gets https:// prepended.
maxItemsPerFeed	integer	0	Cap saved items per feed, applied after filtering. 0 = no limit.
sinceDate	string	""	Keep only items published after this date. 2026-01-01 or a full ISO-8601 timestamp.
keywordFilter	array	[]	Keep only items whose title, summary or categories contain one of these terms. Empty = all.
includeContent	boolean	true	Include the full body in contentText. Off gives a much smaller dataset.
stripHtmlTags	boolean	true	On, bodies are plain text only. Off additionally keeps the original markup in contentHtml.

Filters combine with AND: an item must pass sinceDate and match at least one keyword. Items with no publish date at all are never dropped by sinceDate.

Output example

{
 "requestedUrl": "https://arstechnica.com/",
 "feedUrl": "https://arstechnica.com/feed/",
 "feedTitle": "Ars Technica",
 "feedType": "rss",
 "itemId": "https://arstechnica.com/gadgets/2026/08/ai-isnt-enough-to-protect-social-media-communities-from-ai/",
 "title": "AI isn't enough to protect social media communities from AI",
 "link": "https://arstechnica.com/gadgets/2026/08/ai-isnt-enough-to-protect-social-media-communities-from-ai/",
 "author": "Scharon Harding",
 "publishedAt": "2026-08-06T11:00:44.000Z",
 "updatedAt": null,
 "summary": "Why humans need to moderate humans.",
 "contentText": "Sometimes you have to fight fire with fire. But when it comes to AI slop and hateful content threatening the safety and value of social media platforms …",
 "contentHtml": null,
 "categories": ["AI", "Tech", "discord", "Facebook", "generative ai", "reddit", "social media"],
 "enclosures": [],
 "imageUrl": "https://cdn.arstechnica.net/wp-content/uploads/2026/07/GettyImages-2259634870-1152x648.jpg",
 "scrapedAt": "2026-08-06T11:43:41.586Z"
}

Finding a feed URL

You usually do not have to. Paste the site and let the Actor look:

1. The URL is fetched and parsed as a feed. If it is one, done.
2. If it is an HTML page, its <link rel="alternate" type="application/rss+xml"> tags are read and the first advertised feed is followed. Atom and JSON Feed type values work the same way.
3. If the page advertises nothing, the conventional paths are tried in order: /feed, /rss, /rss.xml, /feed.xml, /atom.xml, /index.xml.

If you would rather be explicit, most publishers link their feed in the page footer, and WordPress sites are almost always at /feed/.

API example

curl -X POST "https://api.apify.com/v2/acts/arman-bd~rss-feed-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "feedUrls": ["https://news.ycombinator.com/rss", "https://github.blog"],
 "maxItemsPerFeed": 25,
 "includeContent": false
 }'

JavaScript example

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('arman-bd/rss-feed-scraper').call({
 feedUrls: ['https://arstechnica.com', 'https://blog.rust-lang.org/feed.xml'],
 sinceDate: '2026-08-01',
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const item of items) console.log(`${item.publishedAt}, ${item.feedTitle}: ${item.title}`);

Limits and behaviour

• Feeds only carry a window. Most publishers expose their 10-50 most recent items and nothing older. This Actor returns what the feed contains; it is not an archive crawler.
• Formats are a mess, and that is handled. RSS 0.91, RSS 1.0 (RDF), RSS 2.0, Atom 1.0 and JSON Feed 1.x all normalise to the same record. CDATA, double-encoded entities, content:encoded, media:* and itunes:* extensions and relative links are all dealt with.
• Dates are normalised. RFC-822 (Thu, 06 Aug 2026 03:56:29 GMT) and ISO-8601 both come out as ISO-8601 UTC. A date the parser cannot read becomes null rather than a wrong guess.
• One failure never kills the run. A dead host, a 404 or a page with no discoverable feed is logged, recorded in RUN_SUMMARY.failures, and the next source is read. The Actor only errors out if every source fails.
• Transient errors are retried. 429 and 5xx get three attempts with backoff, honouring Retry-After when the server sends one.
• One request per second between sources, with a 45-second timeout each.
• Public data only. No authentication, no personal data, no paywall circumvention.

FAQ

Do I need a proxy? No. Proxy configuration is not required to run this Actor.

Can it read a feed behind a login? No. Authenticated and private feeds are out of scope; you will get an "access denied" failure in RUN_SUMMARY.

What if a site has several feeds? Discovery follows the first advertised one. If you want a specific category feed, pass its URL directly, you can pass as many as you like in one run.

Why is contentText the same as summary for some feeds? Because that feed only publishes a summary. Hacker News and most link blogs never send a body; nothing is being dropped.

Why is author null? Plenty of RSS feeds simply omit dc:creator and <author>. The Actor does not guess.

Can I get only new items? Run on a schedule and diff on itemId, or set sinceDate to your last run time. includeContent: false makes those sweeps much cheaper.

Can I integrate it with something else? Yes, Apify API, client libraries, webhooks, scheduled runs, dataset exports (JSON/CSV/Excel) or MCP. Output is structured JSON.