RSS to Markdown: Full-Text Feed Reader for LLM & RAG

Turn RSS feeds into full-text Markdown for LLMs, RAG and newsletters. Give it RSS, Atom or JSON Feed URLs. It returns one record per post with the full article text as clean Markdown, not just the teaser from the feed. Set a State key and run it on a schedule: each run returns only the posts published since the last run, and you pay only for those.

Input	A list of feed URLs (RSS 2.0, RSS 1.0 / RDF, Atom, JSON Feed)
Output	Title, URL, author, date, categories, summary, full article as Markdown, word count, content hash
Price	$0.003 per full-text article from 2026-11-15 ($0.002 until then), $0.0005 per feed-only entry, $0.005 per run. 10 blogs, 30 new posts a day ≈ $0.10/day
Speed	20 full-text articles from two feeds in about 1 minute on 256 MB. No browser

{
    "feed_title": "Apify Blog",
    "title": "Apify vs. Exa: which gives AI agents better access to web data?",
    "url": "https://blog.apify.com/apify-vs-exa-comparison",
    "published": "2026-09-21T10:39:33.000Z",
    "author": "Egop Gogo-Job",
    "content_markdown": "An [AI agent](https://en.wikipedia.org/wiki/AI_agent) can find the right webpage and still miss the data it needs.\n\nSome details are embedded in a page’s code...",
    "content_source": "page",
    "word_count": 2935
}

Jump to the task you have:

1. Get the full article text from an RSS feed, not just the summary
2. Get only new posts since the last run
3. Load blog posts into a RAG index, vector database or LLM wiki
4. Build a daily digest or newsletter in n8n, Make or Zapier
5. Let an AI agent read a feed
6. What this Actor cannot do

1. Get the full article text from an RSS feed, not just the summary

Most feeds carry only a short <description>. This Actor opens each post's page over plain HTTP and keeps only the article: it tries known layouts first (Ghost, WordPress, Substack, beehiiv, Medium-style, docs sites), then falls back to "the block with the most text", and removes navigation, share bars, related posts, newsletter boxes, comments and cookie banners. Headings, links, lists, code blocks and images are kept as GitHub-flavoured Markdown.

{ "feedUrls": ["https://blog.apify.com/rss/"], "maxItemsPerFeed": 10 }

If a feed already contains the whole post (content:encoded, Atom content, JSON content_html), that is used when the page cannot be read. Each record says where the text came from in content_source: page, feed or null (summary only).

If the wrong part of the page is picked up, set contentSelector to the article container (for example .post-content or article .entry) and list stray blocks in removeSelectors.

2. Get only new posts since the last run

Set State key to any name, for example rss-state-ai-blogs, and create an Apify Schedule (daily or hourly) with that input. The Actor remembers every URL it has exported in a named key-value store, so the next run skips them and returns only new posts. Skipped posts are not charged.

{
    "feedUrls": ["https://blog.apify.com/rss/", "https://simonwillison.net/atom/everything/"],
    "stateKey": "rss-state-ai-blogs",
    "sinceDays": 7,
    "maxItemsPerFeed": 50
}

• The same post behind different links counts once: utm_*, fbclid and other tracking parameters and #fragments are removed before comparing.
• sinceDays keeps the first run small (otherwise a feed with 50 old posts returns all 50 once).
• Use a different State key per project. Deleting the key-value store resets it.
• SUMMARY.json in the run's key-value store shows, per feed, how many posts were seen, kept, skipped as already exported, skipped as too old, and any error.

3. Load blog posts into a RAG index, vector database or LLM wiki

Each record is one document: content_markdown is the text to chunk and embed; title, url, author, published and categories are the metadata to store with it. Two hashes help you keep the index clean:

• url_hash (SHA-256 of the normalised URL) is a stable document ID. Use it as the primary key so re-runs never create duplicates.
• content_hash (SHA-256 of the Markdown) tells you whether the text you stored differs from what you have now.

Read the dataset with the Apify API (/v2/datasets/{id}/items?format=json) or the Apify integrations for Pinecone, Qdrant, Weaviate and others. word_count lets you skip very short items (link posts, podcasts) before embedding.

4. Build a daily digest or newsletter in n8n, Make or Zapier

Schedule the Actor with a State key (section 2), then in n8n / Make / Zapier use the Apify app's "run finished" trigger and read the run's dataset. Every item is a new post, so the digest step needs no dedupe logic of its own. summary is a plain-text teaser (up to 1,000 characters) for the email; content_markdown is there if you want an LLM to write the summary.

5. Let an AI agent read a feed (MCP)

With the Apify MCP server, an agent (Claude, Cursor, ChatGPT and others) can call this Actor by name with a feed URL and read the Markdown from the dataset. The dataset schema describes every field, so the agent knows what it gets back. Keep maxItemsPerFeed low (5 to 10) for interactive use.

What this Actor cannot do

Read this before your first run. Nothing here fails the whole run: a broken feed or page is reported and the rest continues.

Limit	What happens	What to do
JavaScript-only pages (articles rendered in the browser)	The page HTML has no article text. The record keeps the feed's content or summary, extracted: false, extraction_error: "no article text found on page", charged at the lower $0.0005 rate	Use a feed that embeds full content, or a browser-based crawler for that site
Paywalls and logins	Only what a logged-out visitor sees is extracted (often the first paragraphs). No cookies or credentials can be passed	None. Private feeds that need a token in a header are not supported
Sites that block bots (Cloudflare challenge, 403)	The page fails after 2 retries; the record keeps the feed data with extraction_error: "page fetch failed: ..."	Turn on Apify Proxy in proxyConfiguration
robots.txt disallows the article	The page is not fetched (on by default); feed data is stored	Set respectRobotsTxt: false only if you have the right to fetch those pages
Older posts than the feed contains	A feed usually lists only the newest 10 to 50 posts. There is no archive crawling	Run on a schedule from now on, or use a sitemap/docs crawler for the back catalogue
A post is edited after export	Dedupe is by URL, so an edited post is not exported again	Compare content_hash yourself if you need edits
A web page URL instead of a feed URL	That feed fails with an error that lists the feed URLs the page advertises (<link rel="alternate">), or suggests /feed, /rss, /atom.xml	Paste the suggested feed URL
Podcasts, PDFs, media	Enclosures are ignored. Links ending in .pdf, .mp3, .mp4, images, .xml, .json and similar are stored as feed data only. No transcription	—
Rate limits	Feeds: 30 s timeout, 2 retries with back-off on 429 / 5xx, Retry-After is honoured. Pages: 45 s timeout, 2 retries, 5 in parallel by default	Lower maxConcurrency for small sites that throttle
Very large state	The State key remembers the 100,000 most recent URLs; older ones are forgotten and could be exported again if they reappear in a feed	Split very large projects across several State keys
Budget cap	With Max total charge per run set, the run stops cleanly before going over it. Posts not exported are not saved to the state, so the next run picks them up	Raise the cap or lower maxItemsPerFeed

Input

Field	Type	Default	Notes
feedUrls	string[]	required	RSS / Atom / JSON Feed URLs.
maxItemsPerFeed	integer	50	Newest entries kept per feed after filtering and dedupe. Caps cost. Up to 5,000.
fetchFullContent	boolean	true	Fetch each entry's page and extract the article. Off = feed data only.
sinceDays	integer	0	0 = no date filter. Undated entries are always kept.
stateKey	string	""	Name of a key-value store that remembers exported URLs between runs, e.g. rss-state-newsletter.
contentSelector	string	""	CSS selector of the article body. Empty = auto-detect.
removeSelectors	string[]	[]	Extra selectors stripped from the article.
respectRobotsTxt	boolean	true	Skip article URLs disallowed by robots.txt (feed data is still stored).
maxConcurrency	integer	5	Parallel article requests.
proxyConfiguration	object	no proxy	Enable Apify Proxy only if a site blocks you.

Output fields

One dataset item per entry. Besides the fields in the example above: feed_url, feed_format (rss, rdf, atom, json), id, updated, categories, summary, content_hash, url_hash, page_title, lang, image, extracted, extraction_error, fetched_at.

Field	Meaning
content_source	page (article extracted from the entry URL), feed (converted from the feed's embedded HTML) or null (summary only).
extracted	true only when the body came from the page. This decides which billing event applies.
extraction_error	Why the page was not used: fetch failed, robots.txt, non-HTML URL, or no article text found. The entry is still stored with feed data.
content_hash	SHA-256 of content_markdown (or the summary), for change detection.
url_hash	SHA-256 of the normalised URL; the dedupe key stored under State key.

With a State key, the named store holds SEEN_URLS ({ hashes: { <url_hash>: <fetched_at> }, updatedAt }). It is saved at the end of the run and also when the platform migrates or aborts the run, so a restarted run does not export and charge the same posts twice.

Pricing

Pay-per-event. Every stored entry is charged exactly once, at one of two rates. Platform usage (compute, storage) is included.

Event	Price	When
article-extracted	$0.003 per entry ($3 per 1,000) from 2026-11-15 05:03 UTC; $0.002 until then	The article body was fetched and extracted from the entry's page (extracted: true).
entry-exported	$0.0005 per entry ($0.50 per 1,000)	Feed data only: fetchFullContent off, feed-embedded content, non-HTML links, robots.txt skips or failed pages.
Actor start	$0.005 per run	Apify's standard start event; covers the first seconds of compute.

Entries skipped by the date filter or the State key are never charged.

Verified

Local run on 2026-09-22 (Apify SDK 3.7.2, Crawlee 3.18.1), prefill input (https://blog.apify.com/rss/ + https://hnrss.org/frontpage, 10 per feed): 2 feeds parsed, 20 entries stored, 20 articles extracted, 0 failed pages, 71.8 s (one page timed out once and succeeded on retry). Pay-per-event path tested with a State key: the run stopped cleanly at the budget, and a second run with the same key skipped the 5 exported posts and returned the next 5. npm test runs 23 unit tests (all feed formats on real fixtures, URL normalisation, extraction on a real Ghost page, density fallback, the web-page-instead-of-feed message). The platform runs the prefill input daily as an automated check.

Changelog

• 0.1.5 (2026-10-04): a web page URL in feedUrls now fails with the feed URL(s) the page advertises; the dedupe state is also saved on migration / abort (no double export after a platform restart). README rewritten by task, with a limits table.
• 0.1.1 (2026-09-23): beehiiv newsletters now return the whole post body (previously only one content block was extracted).
• 0.1.0 (2026-09-22): initial release. RSS 2.0 / RDF / Atom / JSON Feed, full-text extraction, date window, cross-run dedupe via named key-value store, pay-per-event billing with two events.

Support

Open an issue on the Actor's Issues tab. Issues are answered within one business day; bug fixes ship as new builds without changing the input schema.

Made by Vellum Labs.