RSS to Markdown (full-text feed reader)

Turn any RSS, Atom or JSON Feed into clean, LLM-ready Markdown. Give the Actor a list of feed URLs and it returns one record per entry with the feed metadata (title, link, author, date, categories, summary) plus the full article extracted from the entry's page as GitHub-flavoured Markdown, with word count and content hash. Remember what was already exported with a state key, and scheduled runs return only new entries.

Most feed readers on Apify Store stop at the feed XML. This one fetches the articles:

• Full text, not teasers. Each entry's page is fetched (plain HTTP, no browser) and the main article is extracted with a readability-style pipeline: known CMS selectors (Ghost, WordPress, Substack, Medium-style, docs frameworks), then a text-density fallback, then chrome removal (navigation, share bars, related posts, newsletter boxes, comments, cookie banners). Feeds that embed the full post (content:encoded, Atom content, JSON content_html) are used directly when fetching is off or the page fails.
• Every feed flavour. RSS 2.0, RSS 1.0 / RDF, Atom 1.0 and JSON Feed 1.x, detected automatically. Namespaced fields (dc:creator, content:encoded, dc:date) are handled.
• Dedupe across runs. Set State key and the Actor keeps a hash of every exported URL in a named key-value store; the next run skips them. Tracking parameters (utm_*, fbclid, ...) and fragments are stripped before hashing, so the same article behind different links is stored once.
• Date window. sinceDays keeps only entries newer than N days.
• Cheap and fast. 20 entries from two feeds, with full-text extraction, in about a minute on 256 MB.

Who is it for

• LLM and RAG pipelines that want article bodies, not <description> snippets.
• Newsletter and digest builders (n8n, Make, Zapier via the Apify integrations) that need "new since last run" semantics.
• Knowledge bases: push new posts from a set of blogs into Obsidian, Notion, a Claude Code wiki or a vector index on a schedule.
• AI agents using the Apify MCP server: call the Actor with a feed URL and read the Markdown from the dataset.

Input

Field	Type	Default	Notes
feedUrls	string[]	required	RSS / Atom / JSON Feed URLs.
maxItemsPerFeed	integer	50	Newest entries kept per feed after filtering and dedupe. Caps cost.
fetchFullContent	boolean	true	Fetch each entry's page and extract the article. Off = feed data only.
sinceDays	integer	0	0 = no date filter. Undated entries are always kept.
stateKey	string	""	Name of a key-value store that remembers exported URLs between runs, e.g. rss-state-newsletter.
contentSelector	string	""	CSS selector of the article body. Empty = auto-detect.
removeSelectors	string[]	[]	Extra selectors stripped from the article.
respectRobotsTxt	boolean	true	Skip article URLs disallowed by robots.txt (feed data is still stored).
maxConcurrency	integer	5	Parallel article requests.
proxyConfiguration	object	no proxy	Enable Apify Proxy only if a site blocks you.

Example input:

{
    "feedUrls": ["https://blog.apify.com/rss/", "https://hnrss.org/frontpage"],
    "maxItemsPerFeed": 10,
    "fetchFullContent": true,
    "stateKey": "rss-state-demo"
}

Output

One dataset item per entry:

{
    "feed_url": "https://blog.apify.com/rss/",
    "feed_title": "Apify Blog",
    "feed_format": "rss",
    "id": "6aa24918b98a56000133a386",
    "title": "Apify vs. Exa: which gives AI agents better access to web data?",
    "url": "https://blog.apify.com/apify-vs-exa-comparison",
    "published": "2026-09-21T10:39:33.000Z",
    "updated": "",
    "author": "Egop Gogo-Job",
    "categories": ["AI agents", "Tool comparisons", "MCP"],
    "summary": "Choose your AI agent’s web tools based on actual evidence. Compare what Apify and Exa returned, what they missed, how long each task took, and how much it cost.",
    "content_markdown": "An [AI agent](https://en.wikipedia.org/wiki/AI_agent) can find the right webpage and still miss the data it needs.\n\nSome details are embedded in a page’s code...",
    "content_source": "page",
    "word_count": 2935,
    "content_hash": "e329100b4488efce1905f0d84a97522cb6806a8737553b1536b2ab35026c642c",
    "url_hash": "d076f53eee76cd7b6fce13af8f43862fa3e10bdfa608b55e2020fd074db27af3",
    "page_title": "How much web data can your AI agent unlock with Apify or Exa?",
    "lang": "en",
    "image": "https://storage.ghost.io/.../Apify_vs_EXA.webp",
    "extracted": true,
    "extraction_error": null,
    "fetched_at": "2026-09-22T14:41:37.157Z"
}

Field	Meaning
content_source	page (article extracted from the entry URL), feed (converted from the feed's embedded HTML) or null (summary only).
extracted	true only when the body came from the page. This decides which billing event applies.
extraction_error	Why the page was not used: fetch failed, robots.txt, non-HTML URL, or no article text found. The entry is still stored with feed data.
content_hash	SHA-256 of content_markdown (or the summary), for change detection.
url_hash	SHA-256 of the normalised URL; the dedupe key stored under State key.

SUMMARY.json in the key-value store lists per-feed statistics (entries seen, kept, skipped as already exported, skipped as too old, errors) and run totals. With a State key, the named store holds SEEN_URLS ({ hashes: { <url_hash>: <fetched_at> }, updatedAt }, capped at 100,000 newest entries).

Pricing

This Actor uses pay-per-event pricing. Every stored entry is charged exactly once, at one of two rates:

Event	Price	When
article-extracted	$0.002 per entry ($2 per 1,000)	The article body was fetched and extracted from the entry's page (extracted: true).
entry-exported	$0.0005 per entry ($0.50 per 1,000)	Feed data only: fetchFullContent off, feed-embedded content, non-HTML links, robots.txt skips or failed pages.
Actor start	$0.005 per run	Apify's standard start event; covers the first seconds of compute.

A daily run over 10 blogs that yields 30 new full-text articles costs about $0.07. Entries skipped by the date filter or the dedupe state are never charged. Set Max total charge per run to hard-cap spend; the Actor stops gracefully before exceeding it. Platform usage (compute, storage) is included in the event prices.

Tips

• Scheduling: create a Schedule with a fixed stateKey; each run stores only entries not seen before. Use a different key per project.
• First run of a big archive: combine sinceDays: 30 with maxItemsPerFeed so the first run does not export years of history.
• Wrong body picked up: pass contentSelector (e.g. .post-content, article .entry) and add stray blocks to removeSelectors.
• Paywalled or JavaScript-only sites: the Actor fetches HTML over HTTP and does not run JavaScript; such entries fall back to the feed's content or summary and are charged at the lower entry-exported rate.
• Podcasts / media feeds: enclosure links are ignored; the entry's link is used. Non-HTML links (.pdf, .mp3, ...) are stored as feed data only.

Limits

• Feeds are fetched with a 30 s timeout and two retries (back-off on 429/5xx). A feed that fails is reported in SUMMARY.json and does not fail the run.
• Article pages: 45 s navigation timeout, 2 retries, respectRobotsTxt on by default.
• maxItemsPerFeed up to 5,000; dedupe state holds the 100,000 most recent URLs per state key.

Verified

Local run on 2026-09-22 with apify run (Apify CLI 1.10.0, Apify SDK 3.7.2, Crawlee 3.18.1, Node 24), prefill input (https://blog.apify.com/rss/ + https://hnrss.org/frontpage, maxItemsPerFeed: 10, fetchFullContent: true, concurrency 5):

• 2 feeds parsed (RSS 2.0, 15 and 20 entries), 20 entries stored, 20 articles extracted from 20 page requests (20 succeeded, 0 failed), runtime 71.8 s (one page timed out at 45 s and succeeded on retry; without it about 25 s). Word counts 274 to 5,116 per article; authors and publish dates came from the feed for all 20.
• Output spot-checked: Ghost posts (blog.apify.com) extracted from .post-content with headings, links and images, no related-posts / newsletter / share chrome; HN front-page links (news sites, personal blogs, a forum) extracted via known selectors or the text-density fallback.
• Pay-per-event path tested with ACTOR_TEST_PAY_PER_EVENT=1 ACTOR_MAX_TOTAL_CHARGE_USD=5 (local test price $1/event) and stateKey: rss-state-test: 5 entries stored and charged as article-extracted, then "Run budget will be exhausted by the next entry - stopping gracefully". A second run with the same state key reported skippedSeen: 5 and exported the next 5 entries; the named store ended with 10 hashes.
• Unit tests: npm test (19 tests with node --test: format detection, RSS 2.0 / RDF / Atom / JSON Feed parsing on real and synthetic fixtures, URL normalisation and hashing, article extraction on a real Ghost page, density fallback, explicit selector, feed-HTML conversion). eslint and apify validate-schema pass.

Changelog

• 0.1.1 (2026-09-23): beehiiv newsletters now return the whole post body (previously only one content block was extracted).
• 0.1.0 (2026-09-22): initial release. RSS 2.0 / RDF / Atom / JSON Feed, full-text extraction, date window, cross-run dedupe via named key-value store, pay-per-event billing with two events.

Support

Open an issue on the Actor's Issues tab. Issues are answered within one business day; bug fixes ship as new builds without changing the input schema.

Made by Vellum Labs.