# RSS & Atom Feed Scraper — Any Feed to Clean JSON (Bulk) (`yadroo/rss-to-json`) Actor

Turn any RSS, Atom or JSON Feed (or a website URL, feed auto-discovered) into clean JSON: title, link, summary, content, author, date, categories, images. Bulk feeds, whole-word keyword include/exclude, date window, dedupe, sort, only-new monitoring, optional full article text. No API key.

- **URL**: https://apify.com/yadroo/rss-to-json.md
- **Developed by:** [Samat Makatov](https://apify.com/yadroo) (community)
- **Categories:** News, Automation, Integrations
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 result items

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## RSS & Atom Feed Scraper: any feed to clean JSON, with filters, monitoring and full article text

Turn hundreds of RSS 2.0, RSS 1.0/RDF, Atom and JSON Feed sources into one clean, deduplicated, date-sorted JSON dataset. Give it feed URLs or just website URLs (the feed is auto-discovered), filter by whole-word keywords and date window, output only items you have not seen before, and optionally pull the full article text. No API key, no proxy, no browser.

### Use cases

- **Media and brand monitoring**: watch news and industry feeds for your brand, competitors or tickers, scheduled hourly with `onlyNew`.
- **Market and investment signals**: collect every "funding", "acquisition", "IPO" story from startup and business news in the last 24 h.
- **AI agents and RAG pipelines**: feed an LLM fresh, deduplicated articles with full text and reading time.
- **Content curation and newsletters**: merge dozens of blogs into one newest-first list with images and summaries.
- **Lead generation**: spot companies announcing hires, expansions or tenders in regional news feeds.
- **Alerting**: send new matches to Slack / Telegram / email via an Apify integration or webhook.

### Input

| Field | Type | Default | Allowed values / notes |
|---|---|---|---|
| `urls` | string\[] | required | Feed URLs or website URLs (`https://` added if missing) |
| `discoverFeeds` | boolean | true | Find the feed on HTML pages via `<link rel="alternate">` and common paths |
| `includeKeywords` | string\[] | none | Whole words/phrases, case-insensitive; `invest*` = prefix |
| `excludeKeywords` | string\[] | none | Same rules; any match drops the item |
| `keywordMatch` | string | `any` | `any`, `all` |
| `matchIn` | string | `titleSummary` | `title`, `titleSummary`, `all` (title, summary, feed content, categories, author) |
| `query` | string | none | Legacy: all space-separated words must appear in title/summary (ignored when `includeKeywords` is set) |
| `since` | string | none | ISO date/datetime or relative `30m`, `24h`, `7d`, `2w` |
| `until` | string | none | ISO date/datetime or relative; date-only = end of that day (UTC) |
| `sinceHours` | integer | 0 | Legacy: last N hours (0 = off) |
| `onlyNew` | boolean | false | Output only items not output by previous runs with the same `stateKey` |
| `stateKey` | string | derived from URLs | Name of the monitor's memory |
| `limitPerFeed` | integer | 50 | 1–1000, after filters and sorting |
| `maxItems` | integer | 1000 | 1–10 000 overall |
| `sortBy` | string | `publishedDesc` | `publishedDesc`, `publishedAsc`, `feedOrder` |
| `dedupeBy` | string | `url` | `url`, `id`, `title`, `none` |
| `fetchFullContent` | boolean | false | Fetch each item's page and extract article text |
| `maxFullContentChars` | integer | 20 000 | 500–200 000 |
| `fullContentConcurrency` | integer | 3 | 1–10 |
| `summaryMaxChars` | integer | 1000 | 0–20 000 (0 = no limit) |
| `contentMaxChars` | integer | 5000 | 0–200 000 (0 = no limit) |
| `includeContentHtml` | boolean | false | Add raw `contentHtml` |
| `includeErrors` | boolean | false | Also push a `{feedUrl, httpStatus, error}` row per failed feed (dataset view "Failed feeds"; such rows are charged). Off by default: failed feeds are listed free in `SUMMARY.errors` and the status message |
| `fields` | string\[] | all | Keep only these output fields |
| `userAgent` | string | feed-reader UA | Override the User-Agent. By default the actor identifies as a feed reader and, if a site answers 401/403, retries once with a regular browser User-Agent; a custom value is used as-is (no fallback). |

### Reference

#### Supported formats

| Format | Detected by | Notes |
|---|---|---|
| RSS 2.0 | `<rss>` | `content:encoded`, `dc:creator`, `media:*`, `enclosure`, iTunes podcast tags |
| RSS 1.0 / RDF | `<rdf:RDF>` | `dc:date`, `dc:creator` |
| Atom 1.0 | `<feed>` | `rel="alternate"` link, `published`/`updated`, `author/name`, `category term` |
| JSON Feed 1.0/1.1 | `version: jsonfeed.org` | `content_html`/`content_text`, `authors`, `attachments`, `tags` |
| HTML page | `<html>` | Feed discovered from `<link rel="alternate" type="application/rss+xml|atom+xml|feed+json">`, feed-looking links, then `/feed`, `/rss`, `/feed.xml`, `/rss.xml`, `/atom.xml`, `/index.xml`, `/feeds/posts/default`, `/blog/feed`, `/blog/rss`, `/?feed=rss2` |

Non-UTF-8 feeds (e.g. `windows-1251`, `koi8-r`) are decoded using the XML declaration or `Content-Type` charset.

#### Keyword rules

| Keyword | Matches | Does not match |
|---|---|---|
| `ai` | "new AI model" | "StarCraft", "said" |
| `interest rate` | "interest rate hike" | "interest rates" |
| `interest rate*` | "interest rates" | |
| `invest*` | "investor", "investing" | "reinvest" |
| `нефть` | "цены на нефть" | |

#### Time windows

| Value | Meaning |
|---|---|
| `24h`, `7d`, `2w`, `30m` | Relative to run start |
| `2026-09-01` | Since: 00:00 UTC that day. Until: 23:59:59.999 UTC that day |
| `2026-09-01T12:00:00+05:00` | Exact instant |

### Examples

**Brand and competitor monitoring (schedule hourly)**

```json
{
  "urls": ["https://techcrunch.com", "https://www.theverge.com/rss/index.xml", "https://feeds.bbci.co.uk/news/business/rss.xml"],
  "includeKeywords": ["kaspi", "revolut", "wise", "monzo"],
  "matchIn": "all",
  "onlyNew": true,
  "stateKey": "fintech-competitors"
}
```

**Funding and M\&A signals from the last 24 hours**

```json
{
  "urls": ["https://techcrunch.com/feed/", "https://news.crunchbase.com/feed/", "https://hnrss.org/frontpage"],
  "includeKeywords": ["raises", "funding", "series a", "series b", "acquires", "acquisition"],
  "excludeKeywords": ["podcast", "webinar"],
  "since": "24h",
  "sortBy": "publishedDesc"
}
```

**Fresh articles with full text for an AI digest**

```json
{
  "urls": ["https://hnrss.org/frontpage", "https://blog.apify.com/rss/"],
  "includeKeywords": ["ai", "llm", "agent*"],
  "since": "7d",
  "limitPerFeed": 10,
  "maxItems": 25,
  "fetchFullContent": true,
  "maxFullContentChars": 8000
}
```

**Regional news archive for a date range**

```json
{
  "urls": ["https://tengrinews.kz/news.rss"],
  "since": "2026-09-01",
  "until": "2026-09-07",
  "sortBy": "publishedAsc",
  "limitPerFeed": 500,
  "fields": ["publishedAt", "title", "summary", "url", "image"]
}
```

**Newsletter curation across many blogs, one story once**

```json
{ "urls": ["https://github.blog/feed/", "https://blog.cloudflare.com/rss/", "https://aws.amazon.com/blogs/aws/feed/"], "limitPerFeed": 5, "dedupeBy": "title" }
```

### Output

Real item (trimmed) from a run with `includeKeywords: ["ai","openai","nvidia"]`, `since: "7d"` and `fetchFullContent: true`:

```json
{
  "id": "4ddde2983899dca3",
  "feedUrl": "https://feeds.bbci.co.uk/news/business/rss.xml",
  "feedTitle": "BBC News",
  "feedLink": "https://www.bbc.co.uk/news/business",
  "feedFormat": "rss",
  "title": "AI staff 'genuinely frightened' for humanity's future, ex-Anthropic researcher tells BBC",
  "url": "https://www.bbc.co.uk/news/articles/c1kx0gyje9wo?at_medium=RSS&at_campaign=rss",
  "summary": "It comes as the AI firm's boss has called for the technology's development to be slowed down, citing \"serious\" risks.",
  "content": null,
  "author": null,
  "publishedAt": "2026-09-13T07:06:41.000Z",
  "updatedAt": null,
  "categories": [],
  "image": "https://ichef.bbci.co.uk/ace/standard/240/cpsprodpb/09f1/live/aaa670e0-af38-11f1-a540-61c3f7fc4e6c.jpg",
  "enclosures": [],
  "commentsUrl": null,
  "language": "en-gb",
  "guid": "https://www.bbc.co.uk/news/articles/c1kx0gyje9wo#0",
  "feedPosition": 0,
  "keywordsMatched": ["ai"],
  "fullContent": "By Brandon Livesay\n\nPublished\n13 September 2026, 02:15 BST\n\n…An artificial intelligence researcher who quit AI firm Anthr…",
  "fullContentWordCount": 1059,
  "readingTimeMin": 5,
  "canonicalUrl": "https://www.bbc.co.uk/news/articles/c1kx0gyje9wo",
  "siteName": "BBC News",
  "fullContentError": null,
  "fetchedAt": "2026-09-13T08:14:16.114Z"
}
```

| Field | Description |
|---|---|
| `id` | Stable 16-hex id (hash of feed URL + GUID/link), used by `onlyNew` |
| `feedUrl`, `feedTitle`, `feedLink`, `feedFormat`, `sourceUrl` | Feed metadata; `sourceUrl` = the website URL you gave when the feed was discovered |
| `title`, `url`, `guid`, `feedPosition` | Item identity; relative links are made absolute |
| `summary`, `content`, `contentHtml` | Plain-text summary and feed-embedded content (truncated), raw HTML on request |
| `author`, `publishedAt`, `updatedAt`, `language` | ISO 8601 UTC dates |
| `categories`, `image`, `enclosures`, `commentsUrl` | Tags, best image (media/enclosure/first `<img>`), podcast/media files |
| `keywordsMatched` | Include keywords found in the item |
| `readingTimeMin` | From full text if fetched, else from feed content |
| `fullContent`, `fullContentWordCount`, `canonicalUrl`, `siteName`, `pageDescription`, `fullContentError` | With `fetchFullContent` |
| `fetchedAt` | Run timestamp |
| `error`, `httpStatus` | Only on failed-feed rows, which appear only with `includeErrors: true` (`feedUrl` + `httpStatus` + `error`): the error says what failed, what was already tried and what to do (e.g. 403 = bot protection / cloud-IP block, 404 = feed moved — pass the homepage to auto-discover) |

The `SUMMARY` record in the key-value store lists every feed with its resolved URL, format, items in feed, matches, saved count and error, plus `errors` (`[{input, error, httpStatus}]` for failed feeds) and `errorCount`. The run's status message says how many feeds failed. Failed feeds cost nothing.

### Use it from code / agents

```bash
curl -X POST "https://api.apify.com/v2/acts/yadroo~rss-to-json/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://techcrunch.com"],"includeKeywords":["funding"],"since":"24h"}'
```

```js
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('yadroo/rss-to-json').call({ urls: ['https://hnrss.org/frontpage'], includeKeywords: ['llm'], onlyNew: true, stateKey: 'hn-llm' });
const { items } = await client.dataset(run.defaultDatasetId).listItems();
```

```python
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("yadroo/rss-to-json").call(run_input={"urls": ["https://github.blog/feed/"], "since": "7d", "fetchFullContent": True})
items = client.dataset(run["defaultDatasetId"]).list_items().items
```

MCP: add `https://mcp.apify.com` to Claude / Cursor / any MCP client and call the `yadroo/rss-to-json` tool with the same JSON input.

### Pricing

Pay per event: **$0.001 per run start + $0.001 per dataset row** (feed item). Failed feeds are free — they are reported in `SUMMARY` and the status message, not as dataset rows (unless you turn on `includeErrors`).
Typical runs: 25-item digest ≈ $0.026; 500 items from 10 feeds ≈ $0.501; an hourly `onlyNew` monitor that finds 5 new items per run ≈ $0.006 per run.

### Limits & FAQ

- **Politeness**: feeds are fetched 4 at a time, articles 3 at a time (configurable) with a short pause; 429/5xx are retried with backoff (honouring short `Retry-After`). 4xx errors are not retried.
- **Freshness**: every run reads the live feed; how far back you can go depends on how many items the publisher keeps in the feed (often 10–100). This is not a historical archive.
- **Dates**: items without a parseable date are kept unless `since` is set; they sort last.
- **Full content**: extraction is heuristic (no browser): works on most news sites and blogs; paywalled, JavaScript-only or bot-protected pages return `fullContentError` (e.g. `HTTP 403`). Respect the publisher's terms when republishing text.
- **Failures**: a broken feed never stops the others; it is listed in `SUMMARY.errors` / `SUMMARY.feeds` and in the status message (not charged), and the run log ends with a warning naming each failed feed. The run fails only if every feed fails.
- **Blocked feeds (403)**: some publishers filter bots by User-Agent — The Guardian, for example, refuses any User-Agent that mentions "apify" (earlier versions of this actor sent one; fixed). The actor now sends a neutral feed-reader User-Agent and falls back to a browser User-Agent on 401/403. A feed that still returns 403 is blocking the cloud IP range itself; it shows up with `httpStatus: 403` in `SUMMARY.errors` and the status message rather than disappearing silently.
- **Size**: feeds up to 15 MB, article pages up to 4 MB.
- **Roadmap**: OPML import, per-feed custom User-Agent, sitemap-based "feeds" for sites without RSS.

***

Made by **Yadroo**. Sibling actors: [google-news-search](https://apify.com/yadroo/google-news-search), [crypto-news](https://apify.com/yadroo/crypto-news), [hackernews-search](https://apify.com/yadroo/hackernews-search), [youtube-channel-feed](https://apify.com/yadroo/youtube-channel-feed), [arxiv-papers](https://apify.com/yadroo/arxiv-papers), [wikipedia-search](https://apify.com/yadroo/wikipedia-search).

# Actor input Schema

## `urls` (type: `array`):

RSS 2.0, RSS 1.0 (RDF), Atom or JSON Feed URLs — or plain website URLs: the feed is auto-discovered from <link rel="alternate"> and common paths (/feed, /rss.xml, /atom.xml…). Up to a few hundred URLs per run; fetched 4 at a time.

## `discoverFeeds` (type: `boolean`):

If a URL returns an HTML page, look for its feed. Off = HTML pages are reported as errors.

## `includeKeywords` (type: `array`):

Keep only items mentioning these words/phrases (case-insensitive, whole words: "ai" does not match "StarCraft"). Add a trailing \* for prefixes: "invest\*". Matched words are returned in keywordsMatched.

## `excludeKeywords` (type: `array`):

Drop items mentioning any of these words/phrases (same matching rules).

## `keywordMatch` (type: `string`):

Whether an item needs one or all of the include keywords.

## `matchIn` (type: `string`):

Which item text the keyword filters look at. Full article text (fetchFullContent) is fetched after filtering and is not searched.

## `query` (type: `string`):

Space-separated words that must ALL appear in title/summary. Ignored when includeKeywords is set.

## `since` (type: `string`):

ISO date/datetime (2026-09-01, 2026-09-01T12:00:00Z) or relative span back from now: 30m, 24h, 7d, 2w. Items without a parseable date are dropped when set.

## `until` (type: `string`):

ISO date/datetime or relative span. A date-only value includes that whole day (UTC).

## `sinceHours` (type: `integer`):

Only items from the last N hours (0 = no limit). If both are set, the later of since / sinceHours wins.

## `onlyNew` (type: `boolean`):

Monitoring mode: remembers item ids in the named key-value store "rss-to-json-state" and outputs only new items on each scheduled run (last 20 000 ids kept).

## `stateKey` (type: `string`):

Name of the seen-items memory for onlyNew. Default is derived from the URL list; set it explicitly to keep state when you edit the list. Letters, digits, dot, dash, underscore.

## `limitPerFeed` (type: `integer`):

Cap per feed, applied after filters and sorting (so with publishedDesc you get each feed's newest N matches).

## `maxItems` (type: `integer`):

Overall cap after merging feeds (keeps the first N in the chosen sort order).

## `sortBy` (type: `string`):

Order of the merged output. Undated items go last.

## `dedupeBy` (type: `string`):

Removes the same story appearing in several feeds or twice in one feed.

## `fetchFullContent` (type: `boolean`):

Open each output item's link and extract the main article text (plus word count, reading time, canonical URL, og:image/author/date when missing). One extra request per item; paywalled or bot-protected sites return fullContentError.

## `maxFullContentChars` (type: `integer`):

Truncate fullContent to this length.

## `fullContentConcurrency` (type: `integer`):

Parallel article requests. Keep low for single-site feeds to stay polite.

## `summaryMaxChars` (type: `integer`):

Plain-text summary length (0 = no limit).

## `contentMaxChars` (type: `integer`):

Plain-text length of the content embedded in the feed (content:encoded / Atom content). 0 = no limit.

## `includeContentHtml` (type: `boolean`):

Add contentHtml with the feed's original HTML (not truncated).

## `includeErrors` (type: `boolean`):

Off (default): feeds that could not be fetched/parsed are listed free of charge in the SUMMARY record (errors, feeds) and the run's status message. On: also push a {feedUrl, httpStatus, error} row per failed feed — such rows are dataset rows and are charged like items.

## `fields` (type: `array`):

Keep only these fields, e.g. \["publishedAt","feedTitle","title","url"]. Empty = all.

## `userAgent` (type: `string`):

Override the User-Agent header (some feeds serve 403 to unknown bots).

## Actor input object example

```json
{
  "urls": [
    "https://blog.apify.com/rss/",
    "https://techcrunch.com"
  ],
  "discoverFeeds": true,
  "keywordMatch": "any",
  "matchIn": "titleSummary",
  "sinceHours": 0,
  "onlyNew": false,
  "limitPerFeed": 50,
  "maxItems": 1000,
  "sortBy": "publishedDesc",
  "dedupeBy": "url",
  "fetchFullContent": false,
  "maxFullContentChars": 20000,
  "fullContentConcurrency": 3,
  "summaryMaxChars": 1000,
  "contentMaxChars": 5000,
  "includeContentHtml": false,
  "includeErrors": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://blog.apify.com/rss/",
        "https://techcrunch.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("yadroo/rss-to-json").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://blog.apify.com/rss/",
        "https://techcrunch.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("yadroo/rss-to-json").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://blog.apify.com/rss/",
    "https://techcrunch.com"
  ]
}' |
apify call yadroo/rss-to-json --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,yadroo/rss-to-json"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/uTphVdMbzo7HA2ofV/builds/hsCjlkYzjaCqqUy9A/openapi.json