RSS & Atom Feed Scraper — Any Feed to Clean JSON (Bulk)

Turn hundreds of RSS 2.0, RSS 1.0/RDF, Atom and JSON Feed sources into one clean, deduplicated, date-sorted JSON dataset. Give it feed URLs or just website URLs (the feed is auto-discovered), filter by whole-word keywords and date window, output only items you have not seen before, and optionally pull the full article text. No API key, no proxy, no browser.

Use cases

• Media and brand monitoring: watch news and industry feeds for your brand, competitors or tickers, scheduled hourly with onlyNew.
• Market and investment signals: collect every "funding", "acquisition", "IPO" story from startup and business news in the last 24 h.
• AI agents and RAG pipelines: feed an LLM fresh, deduplicated articles with full text and reading time.
• Content curation and newsletters: merge dozens of blogs into one newest-first list with images and summaries.
• Lead generation: spot companies announcing hires, expansions or tenders in regional news feeds.
• Alerting: send new matches to Slack / Telegram / email via an Apify integration or webhook.

Input

Field	Type	Default	Allowed values / notes
urls	string[]	required	Feed URLs or website URLs (https:// added if missing)
discoverFeeds	boolean	true	Find the feed on HTML pages via <link rel="alternate"> and common paths
includeKeywords	string[]	none	Whole words/phrases, case-insensitive; invest* = prefix
excludeKeywords	string[]	none	Same rules; any match drops the item
keywordMatch	string	any	any, all
matchIn	string	titleSummary	title, titleSummary, all (title, summary, feed content, categories, author)
query	string	none	Legacy: all space-separated words must appear in title/summary (ignored when includeKeywords is set)
since	string	none	ISO date/datetime or relative 30m, 24h, 7d, 2w
until	string	none	ISO date/datetime or relative; date-only = end of that day (UTC)
sinceHours	integer	0	Legacy: last N hours (0 = off)
onlyNew	boolean	false	Output only items not output by previous runs with the same stateKey. An item is remembered only once it is stored: items a run did not store (spending limit, timeout) come in the next run
stateKey	string	derived from URLs	Name of the monitor's memory
limitPerFeed	integer	50	1–1000, after filters and sorting
maxItems	integer	1000	1–10 000 items overall (failed-feed rows of includeErrors are not counted)
sortBy	string	publishedDesc	publishedDesc, publishedAsc, feedOrder
dedupeBy	string	url	url, id, title, none. The copy from the feed listed first is kept; an item repeated with the same feed and GUID is output once even with none
fetchFullContent	boolean	false	Fetch each item's page and extract article text
maxFullContentChars	integer	20 000	500–200 000
fullContentConcurrency	integer	3	1–10 sites read at once; one site still gets one request at a time, at most one per second
summaryMaxChars	integer	1000	0–20 000 (0 = no limit)
contentMaxChars	integer	5000	0–200 000 (0 = no limit)
includeContentHtml	boolean	false	Add raw contentHtml
includeErrors	boolean	false	Also push a {feedUrl, httpStatus, error} row per failed feed (dataset view "Failed feeds"; such rows are charged, not counted in maxItems). Off by default: failed feeds are listed free in SUMMARY.errors and the status message
fields	string[]	all	Keep only these output fields, in your order; id is always kept (first, unless you place it). Another letter case or a near miss (titel) is read as the column, an unknown name is left out — both reported in the status and SUMMARY.inputCorrections; if no name is a column the run stops with the list of valid names. A column this run's item rows cannot have — the full-text columns without fetchFullContent, keywordsMatched without includeKeywords/query, sourceUrl with discoverFeeds off, error/httpStatus (failed-feed rows only; those always keep all their columns) — is reported in SUMMARY.warnings; if every listed column is one of these the run stops instead of billing rows that hold only id
userAgent	string	feed-reader UA	Override the User-Agent. By default the actor identifies as a feed reader and, if a site answers 401/403, retries once with a regular browser User-Agent; a custom value is used as-is (no fallback).

Reference

keywordMatch, matchIn, sortBy and dedupeBy accept only the values listed above; Apify refuses any other value before the run starts.

Supported formats

Format	Detected by	Notes
RSS 2.0	<rss>	content:encoded, dc:creator, media:*, enclosure, iTunes podcast tags
RSS 1.0 / RDF	<rdf:RDF>	dc:date, dc:creator
Atom 1.0	<feed>	rel="alternate" link, published/updated, author/name, category term
JSON Feed 1.0/1.1	version: jsonfeed.org	content_html/content_text, authors, attachments, tags
HTML page	<html>	Feed discovered from `<link rel="alternate" type="application/rss+xml

Non-UTF-8 feeds (e.g. windows-1251, koi8-r) are decoded using the XML declaration or Content-Type charset.

Keyword rules

Keyword	Matches	Does not match
ai	"new AI model"	"StarCraft", "said"
interest rate	"interest rate hike"	"interest rates"
interest rate*	"interest rates"
invest*	"investor", "investing"	"reinvest"
нефть	"цены на нефть"

Time windows

Value	Meaning
24h, 7d, 2w, 30m	Relative to run start
2026-09-01	Since: 00:00 UTC that day. Until: 23:59:59.999 UTC that day
2026-09-01T12:00:00+05:00	Exact instant

Examples

Brand and competitor monitoring (schedule hourly)

{
  "urls": ["https://techcrunch.com", "https://www.theverge.com/rss/index.xml", "https://feeds.bbci.co.uk/news/business/rss.xml"],
  "includeKeywords": ["kaspi", "revolut", "wise", "monzo"],
  "matchIn": "all",
  "onlyNew": true,
  "stateKey": "fintech-competitors"
}

Funding and M&A signals from the last 24 hours

{
  "urls": ["https://techcrunch.com/feed/", "https://news.crunchbase.com/feed/", "https://hnrss.org/frontpage"],
  "includeKeywords": ["raises", "funding", "series a", "series b", "acquires", "acquisition"],
  "excludeKeywords": ["podcast", "webinar"],
  "since": "24h",
  "sortBy": "publishedDesc"
}

Fresh articles with full text for an AI digest

{
  "urls": ["https://hnrss.org/frontpage", "https://blog.apify.com/rss/"],
  "includeKeywords": ["ai", "llm", "agent*"],
  "since": "7d",
  "limitPerFeed": 10,
  "maxItems": 25,
  "fetchFullContent": true,
  "maxFullContentChars": 8000
}

Regional news archive for a date range

{
  "urls": ["https://tengrinews.kz/news.rss"],
  "since": "2026-09-01",
  "until": "2026-09-07",
  "sortBy": "publishedAsc",
  "limitPerFeed": 500,
  "fields": ["publishedAt", "title", "summary", "url", "image"]
}

Newsletter curation across many blogs, one story once

{ "urls": ["https://github.blog/feed/", "https://blog.cloudflare.com/rss/", "https://aws.amazon.com/blogs/aws/feed/"], "limitPerFeed": 5, "dedupeBy": "title" }

Output

Real item (trimmed) from a run with includeKeywords: ["ai","openai","nvidia"], since: "7d" and fetchFullContent: true:

{
  "id": "4ddde2983899dca3",
  "feedUrl": "https://feeds.bbci.co.uk/news/business/rss.xml",
  "feedTitle": "BBC News",
  "feedLink": "https://www.bbc.co.uk/news/business",
  "feedFormat": "rss",
  "title": "AI staff 'genuinely frightened' for humanity's future, ex-Anthropic researcher tells BBC",
  "url": "https://www.bbc.co.uk/news/articles/c1kx0gyje9wo?at_medium=RSS&at_campaign=rss",
  "summary": "It comes as the AI firm's boss has called for the technology's development to be slowed down, citing \"serious\" risks.",
  "content": null,
  "author": null,
  "publishedAt": "2026-09-13T07:06:41.000Z",
  "updatedAt": null,
  "categories": [],
  "image": "https://ichef.bbci.co.uk/ace/standard/240/cpsprodpb/09f1/live/aaa670e0-af38-11f1-a540-61c3f7fc4e6c.jpg",
  "enclosures": [],
  "commentsUrl": null,
  "language": "en-gb",
  "guid": "https://www.bbc.co.uk/news/articles/c1kx0gyje9wo#0",
  "feedPosition": 0,
  "keywordsMatched": ["ai"],
  "fullContent": "By Brandon Livesay\n\nPublished\n13 September 2026, 02:15 BST\n\n…An artificial intelligence researcher who quit AI firm Anthr…",
  "fullContentWordCount": 1059,
  "readingTimeMin": 5,
  "canonicalUrl": "https://www.bbc.co.uk/news/articles/c1kx0gyje9wo",
  "siteName": "BBC News",
  "fullContentError": null,
  "fetchedAt": "2026-09-13T08:14:16.114Z"
}

Field	Description
id	Stable 16-hex id (hash of feed URL + GUID/link), used by onlyNew
feedUrl, feedTitle, feedLink, feedFormat, sourceUrl	Feed metadata; sourceUrl = the website URL you gave when the feed was discovered
title, url, guid, feedPosition	Item identity; relative links are made absolute
summary, content, contentHtml	Plain-text summary and feed-embedded content (truncated), raw HTML on request
author, publishedAt, updatedAt, language	ISO 8601 UTC dates
categories, image, enclosures, commentsUrl	Tags, best image (media/enclosure/first <img>), podcast/media files
keywordsMatched	Include keywords found in the item
readingTimeMin	From full text if fetched, else from feed content
fullContent, fullContentWordCount, canonicalUrl, siteName, pageDescription, fullContentError	With fetchFullContent
fetchedAt	Run timestamp
error, httpStatus	Only on failed-feed rows, which appear only with includeErrors: true (feedUrl + httpStatus + error): the error says what failed, what was already tried and what to do (e.g. 403 = bot protection / cloud-IP block, 404 = feed moved — pass the homepage to auto-discover)

The SUMMARY record in the key-value store lists every feed with its resolved URL, format, items in feed, matches, saved count, error and status (ok, failed, notRead = the run stopped before reading it), plus errors ([{input, error, httpStatus}] for failed feeds), errorCount, feedsNotRead, skippedAsSeen (onlyNew), fullContent (articles requested / fetched / failed / not fetched before a stop), requestsPerHost, inputCorrections, delivered (rows stored) and stoppedBy (limit, timeout or null). The run's status message says how many items were saved, how many feeds failed or were not read, and why the run stopped early. Failed feeds cost nothing.

Use it from code / agents

curl -X POST "https://api.apify.com/v2/acts/yadroo~rss-to-json/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://techcrunch.com"],"includeKeywords":["funding"],"since":"24h"}'

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('yadroo/rss-to-json').call({ urls: ['https://hnrss.org/frontpage'], includeKeywords: ['llm'], onlyNew: true, stateKey: 'hn-llm' });
const { items } = await client.dataset(run.defaultDatasetId).listItems();

from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("yadroo/rss-to-json").call(run_input={"urls": ["https://github.blog/feed/"], "since": "7d", "fetchFullContent": True})
items = client.dataset(run["defaultDatasetId"]).list_items().items

MCP: add https://mcp.apify.com to Claude / Cursor / any MCP client and call the yadroo/rss-to-json tool with the same JSON input.

Pricing

Pay per event: $0.001 per run start + $0.001 per dataset row (feed item). Bronze −10 %, Silver −20 %, Gold and above −30 % on the row price; the start event is the same on every plan; platform usage is included. Failed feeds are free — they are reported in SUMMARY and the status message, not as dataset rows (unless you turn on includeErrors).

You never pay more than your Maximum cost per run: the actor reads it before working, fetches article text only for the rows it pays for, and ends with "Stopped at your spending limit: N rows delivered" — those N rows are in the dataset (with onlyNew, only they are remembered; the rest come in the next run).

Typical runs: 25-item digest ≈ $0.026; 500 items from 10 feeds ≈ $0.501; an hourly onlyNew monitor that finds 5 new items per run ≈ $0.006 per run.

Limits & FAQ

• Politeness: every site gets one request at a time and at most one per second, whatever the number of feeds or fullContentConcurrency — 4 feeds and up to 10 articles are read at once only when they are on different sites. Redirects are paced the same way. 429/5xx are retried after the site's Retry-After (up to 60 s) or a growing backoff, and that pause holds for all of the run's requests to the site; a site that asks to wait longer than 60 s is not retried and shows up in SUMMARY.errors / fullContentError. 4xx errors are not retried.
• Freshness: every run reads the live feed; how far back you can go depends on how many items the publisher keeps in the feed (often 10–100). This is not a historical archive.
• Dates: items without a parseable date are kept unless since is set; they sort last.
• Full content: extraction is heuristic (no browser): works on most news sites and blogs; paywalled, JavaScript-only or bot-protected pages return fullContentError (e.g. HTTP 403). Respect the publisher's terms when republishing text.
• Full-content speed: about one article per second per site at most, so a feed from one site reads up to ~800 articles in the default 15-minute run; articles from several sites go in parallel. Rows are saved article by article, in the chosen sort order, so nothing waits for the end of the run.
• Timeout: a run that nears its timeout stops starting new requests, keeps every row it saved and ends SUCCEEDED with "Stopped before the run timeout: N rows saved"; feeds and articles it did not get to are listed (feedsNotRead, fullContent.notFetched) and are not output — with onlyNew they come in the next run. Raise the run timeout or lower maxItems / limitPerFeed for big full-text jobs.
• Failures: a broken feed never stops the others; it is listed in SUMMARY.errors / SUMMARY.feeds and in the status message (not charged), and the run log ends with a warning naming each failed feed. The run fails only if every feed fails — and then it writes and bills no rows.
• Blocked feeds (403): some publishers filter bots by User-Agent — The Guardian, for example, refuses any User-Agent that mentions "apify" (earlier versions of this actor sent one; fixed). The actor now sends a neutral feed-reader User-Agent and falls back to a browser User-Agent on 401/403. A feed that still returns 403 is blocking the cloud IP range itself; it shows up with httpStatus: 403 in SUMMARY.errors and the status message rather than disappearing silently.
• Size: feeds up to 15 MB, article pages up to 4 MB.
• Roadmap: OPML import, per-feed custom User-Agent, sitemap-based "feeds" for sites without RSS.

---

Made by Yadroo. Sibling actors: google-news-search, crypto-news, hackernews-search, youtube-channel-feed, arxiv-papers, wikipedia-search.