# RSS Feed Aggregator: Merge, Dedupe & Filter Multiple Feeds (`pankovs/multi-feed-rss-digest`) Actor

Merge many RSS, Atom, RDF and JSON feeds into one ranked digest. Near-duplicate stories from different sources are grouped into one row with all source links, keyword include/exclude filters, only-new items on scheduled runs, and an optional webhook.

- **URL**: https://apify.com/pankovs/multi-feed-rss-digest.md
- **Developed by:** [Pankovs](https://apify.com/pankovs) (community)
- **Categories:** News, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## RSS Feed Aggregator: Merge, Dedupe & Filter Multiple Feeds

Turn any number of RSS, Atom, RDF and JSON feeds into one ranked digest where each story appears once, with every source listed.
Filter by keywords, get only new stories on scheduled runs, and send them to a webhook, with no browser and no article scraping.

### Use cases

- A daily industry or competitor digest from 20 to 200 sources without reading the same news five times.
- A Slack or email newsletter pipeline: schedule the actor hourly and send only new stories through the webhook.
- Brand and keyword monitoring across news sites, blogs, Google News and Bing News searches, release feeds and podcasts.
- Feeding an LLM or a database with a deduplicated, ranked stream instead of raw feed items.

### How to use it

1. Paste your feed URLs into **Feed URLs** (RSS, Atom, RDF or JSON Feed, up to 500).
2. Optionally add include and exclude keywords, a webhook URL and a `stateKey` for this digest.
3. Click **Start**, or create an Apify schedule (for example hourly) so each run returns only the stories you have not seen yet.

### Input

```json
{
  "feedUrls": [
    "https://hnrss.org/frontpage",
    "https://lobste.rs/rss",
    "https://feeds.arstechnica.com/arstechnica/index",
    "https://www.theverge.com/rss/index.xml"
  ],
  "maxAgeHours": 24,
  "keywords": ["rust", "open source"],
  "excludeKeywords": ["sponsored"],
  "onlyNew": true,
  "stateKey": "dev-news",
  "webhookUrl": "https://hooks.zapier.com/hooks/catch/123/abc/"
}
```

| Field | Default | Meaning |
|---|---|---|
| `feedUrls` | required | Feed URLs (max 500 per run). |
| `maxAgeHours` | 48 | Skip items older than this. 0 = no age limit. Items without a date are kept. |
| `publishedAfter` | none | Skip items published before this date (`2026-10-01` or `2026-10-01T08:00:00Z`). |
| `maxItemsPerFeed` | 50 | Newest items taken from each feed. |
| `keywords` / `excludeKeywords` | none | Include / exclude words or phrases (title, summary, categories). |
| `onlyNew` | true | Skip stories already returned by an earlier run with the same `stateKey` (remembered 30 days). |
| `stateKey` | `default` | Separate memory per digest, for example per client or topic. |
| `webhookUrl` | none | POST new digest items here as JSON. |
| `includeDuplicates` | false | Add one free `duplicate` row per merged copy, with `dupOf`. |
| `includeContent` | false | Add the full HTML content the feed itself contains. |
| `timeoutSecs` | 20 | Per-feed timeout. |
| `concurrency` | 5 | Feeds fetched in parallel. |

### Output

One dataset row per story, in rank order (real row from a run on Hacker News + Lobsters):

```json
{
  "recordType": "item",
  "rank": 1,
  "title": "Valen's Memory Safety: A New Kind of Borrow Checking",
  "url": "https://verdagon.dev/blog/valen-group-borrowing",
  "source": "Hacker News: Front Page",
  "feedTitle": "Hacker News: Front Page",
  "feedUrl": "https://hnrss.org/frontpage",
  "publishedAt": "2026-10-11T12:06:22.000Z",
  "sourceCount": 2,
  "matchedKeywords": [],
  "dupOf": null,
  "sources": [
    { "source": "Hacker News: Front Page", "feedTitle": "Hacker News: Front Page", "feedUrl": "https://hnrss.org/frontpage", "title": "Valen's Memory Safety: A New Kind of Borrow Checking", "url": "https://verdagon.dev/blog/valen-group-borrowing", "publishedAt": "2026-10-11T12:06:22.000Z" },
    { "source": "Lobsters", "feedTitle": "Lobsters", "feedUrl": "https://lobste.rs/rss", "title": "Valen's Memory Safety: A New Kind of Borrow Checking", "url": "https://verdagon.dev/blog/valen-group-borrowing", "publishedAt": "2026-10-11T12:23:01.000Z" }
  ],
  "summary": "Article URL: https://verdagon.dev/blog/valen-group-borrowing Comments URL: https://news.ycombinator.com/item?id=50042362 Points: 5 # Comments: 3",
  "author": "verdagon",
  "categories": [],
  "imageUrl": null,
  "enclosures": [],
  "guid": "https://news.ycombinator.com/item?id=50042362",
  "fetchedAt": "2026-10-11T13:08:40.249Z"
}
```

A feed that fails adds a row like this (free):

```json
{ "recordType": "feed-error", "feedUrl": "https://example.com/", "finalUrl": "https://example.com/", "status": 200, "error": "Not an RSS, Atom, RDF or JSON feed: the URL returned an HTML page", "fetchedAt": "2026-10-11T13:08:40.249Z" }
```

The run summary (key-value store record `OUTPUT`) has per-feed status (`feeds[]`: format, item count, error) and the
counts `itemsFetched`, `itemsInWindow`, `clusters`, `duplicatesMerged`, `filteredOut`, `alreadySeen`,
`itemsEmitted`, plus the webhook result.

#### Webhook payload

Sent only when the run has at least one new digest item:
`{ "generatedAt", "stateKey", "itemCount", "feedsOk", "feedsFailed", "items": [first 100 item rows], "feedErrors": [...] }`.

### Pricing (pay per event)

| Event | Price | When |
|---|---|---|
| Run started (`apify-actor-start`) | $0.005 | Once per run, charged by Apify when the run starts. |
| `feed-polled` | $0.001 | Per feed fetched and parsed successfully. Broken feeds are free. |
| `digest-item` | $0.001 | Per story in the digest, after merging, filters and only-new. Duplicate rows and error rows are free. |

Example 1, one-off digest: 20 feeds with about 340 stories after merging = $0.005 + 20 x $0.001 + 340 x $0.001 = **$0.365**.

Example 2, hourly monitoring: the same 20 feeds every hour with only-new, about 5 new stories per run =
$0.005 + $0.020 + $0.005 = **$0.03 per run**, about **$21.60 per month** (720 runs).

If you set a maximum charge per run, the actor stops adding stories at the limit and the unpaid ones stay new for the
next run.

### FAQ and limits

**Which feeds does it read?** RSS 2.0, Atom, RDF (RSS 1.0) and JSON Feed. Gzipped feeds, redirects, non-UTF-8 feeds
(for example windows-1251 or ISO-8859-1) and relative links are handled. Give it feed URLs: a homepage URL is reported
as "not a feed" instead of being crawled for feed links.

**How does it detect duplicates?** Two items are the same story when either:

1. their links point to the same page after cleanup (tracking parameters such as `utm_*`, `fbclid` or `ref`, AMP,
   `www.`/`m.`, trailing slash and `#fragment` removed), or
2. their headlines contain the same words after cleanup (case, accents, punctuation, word order, small words such as
   "the" or "with", "Updated:" / "Breaking:" prefixes, and the feed's own name at the end are ignored). Long headlines
   (8+ words) may differ by one extra word. Numbers must match exactly, so "Rust 1.90" and "Rust 1.91" stay separate,
   and headlines more than 7 days apart are never merged by title.

The rules are deliberately strict. A false merge would hide a story from you, while a missed merge only shows a
duplicate. The earliest published copy becomes the main row; set **Also output duplicate rows** to get the others
too, each with `dupOf` pointing to the main story.

**How are stories ranked?** Stories covered by more sources come first, then the newest.

**Does it work with Google News and Bing News?** Yes. The same headline from different publishers ("... - Reuters",
"... - Yahoo") becomes one row, `source` is the publisher (from the feed's `<source>`), `feedTitle` is the feed it
came from, and `sourceCount` counts publishers.

**How do keyword filters match?** Whole words only (`rust` does not match "trust"), case and accent insensitive, any
language. Keywords are searched in the title, summary and categories.

**What does "only new" do?** With the same `stateKey`, each run returns only stories you have not received yet,
including late copies of an old story in another feed. Stories are remembered for 30 days.

**What happens when a feed is broken?** The run continues. Each broken feed adds a free `feed-error` row with the
reason (HTTP 404, timeout, not a feed, rate limited).

**Limits.**

- Feed content only: full article text is not fetched (the actor never visits article pages).
- Up to 500 feeds and 1000 items per feed per run; a feed larger than 20 MB is reported as an error.
- Private and local network addresses are blocked, for feed URLs and for the webhook URL.
- Runs use at most 1 GB of memory. 500 feeds x 50 items (25,000 items) needs under 400 MB; for hundreds of feeds
  with 1000 items each, split the feeds over several runs or tasks.

### More

Real output, prices and changelog: [RSS Feed Aggregator, merge and dedupe feeds](https://pankovs.com/rss-feed-aggregator).

# Changelog

This Actor's version history is a separate document: https://apify.com/pankovs/multi-feed-rss-digest/changelog.md

# Actor input Schema

## `feedUrls` (type: `array`):

RSS 2.0, Atom, RDF (RSS 1.0) or JSON Feed URLs. Gzipped feeds and redirects are handled. A broken feed adds a feed-error row and the run continues.

## `maxAgeHours` (type: `integer`):

Skip items published longer ago than this. 0 = no age limit. Items without a date are kept.

## `publishedAfter` (type: `string`):

Optional. Skip items published before this date or time (e.g. 2026-10-01 or 2026-10-01T08:00:00Z). When both limits are set, the later one wins.

## `maxItemsPerFeed` (type: `integer`):

Newest items taken from each feed (after the age limit).

## `keywords` (type: `array`):

Optional. Keep only stories whose title, summary or categories contain at least one of these words or phrases (whole words, case and accent insensitive).

## `excludeKeywords` (type: `array`):

Optional. Drop stories that contain any of these words or phrases (e.g. sponsored, webinar).

## `onlyNew` (type: `boolean`):

Remember emitted stories and skip them (and their near-duplicates) on later runs with the same state key. Turn off to get the full digest every run.

## `stateKey` (type: `string`):

Name of the memory used by Only new items. Use a different key per digest (e.g. per client or topic). Stories are remembered for 30 days.

## `webhookUrl` (type: `string`):

Optional. When a run produces new digest items, they are POSTed here as JSON (Zapier, Make, n8n or your own endpoint).

## `includeDuplicates` (type: `boolean`):

Add one extra row per merged duplicate (recordType duplicate, with dupOf pointing to the main story). Duplicate rows are free.

## `includeContent` (type: `boolean`):

Add the full HTML content from the feed (content:encoded / Atom content) to each row. Only what the feed itself contains; article pages are never visited.

## `timeoutSecs` (type: `integer`):

A feed that does not answer in time is reported as a feed-error row.

## `concurrency` (type: `integer`):

How many feeds are fetched at the same time.

## Actor input object example

```json
{
  "feedUrls": [
    "https://hnrss.org/frontpage",
    "https://hnrss.org/best",
    "https://lobste.rs/rss"
  ],
  "maxAgeHours": 48,
  "maxItemsPerFeed": 50,
  "keywords": [],
  "excludeKeywords": [],
  "onlyNew": true,
  "stateKey": "default",
  "includeDuplicates": false,
  "includeContent": false,
  "timeoutSecs": 20,
  "concurrency": 5
}
```

# Actor output Schema

## `digest` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "feedUrls": [
        "https://hnrss.org/frontpage",
        "https://hnrss.org/best",
        "https://lobste.rs/rss"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("pankovs/multi-feed-rss-digest").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "feedUrls": [
        "https://hnrss.org/frontpage",
        "https://hnrss.org/best",
        "https://lobste.rs/rss",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("pankovs/multi-feed-rss-digest").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "feedUrls": [
    "https://hnrss.org/frontpage",
    "https://hnrss.org/best",
    "https://lobste.rs/rss"
  ]
}' |
apify call pankovs/multi-feed-rss-digest --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,pankovs/multi-feed-rss-digest"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/XemQm9zrlB3rR1qqu/builds/qAE8lA0sjtmVLwrnd/openapi.json