# RSS Feed Scraper API & Incremental Monitor (`automa-flow/rss-feed-monitor`) Actor

RSS feed scraper API for RSS, Atom and JSON Feed URLs in bulk, plus incremental monitoring for new and updated items. Get normalized records, conditional HTTP checks, per-feed failures, stable fingerprints and webhook-ready events.

- **URL**: https://apify.com/automa-flow/rss-feed-monitor.md
- **Developed by:** [Vadim Bezrukov](https://apify.com/automa-flow) (community)
- **Categories:** News, Automation, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.00 / 1,000 successful feed checks

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## RSS Feed Scraper API & Incremental Monitor

Use this RSS feed scraper API to parse RSS, Atom and JSON Feed URLs and monitor
them for genuinely new or updated items without processing duplicates on every
run.

This Actor turns a feed watchlist into schedule- and webhook-ready events. It
uses each feed's native item IDs, remembers only the last successful bounded
state, and sends standard `If-None-Match` / `If-Modified-Since` validators on
later runs. A publisher that answers `304 Not Modified` costs one small check and
no feed download or parse.

### Quick start: RSS to JSON

```json
{
  "feeds": [
    {"url": "https://news.ycombinator.com/rss", "externalId": "hacker-news"}
  ],
  "mode": "snapshot",
  "outputMode": "all",
  "maxItemsPerFeed": 20
}
```

Every current item becomes one normalized `feed_item` row. The Actor returns
content already present in the feed; it never visits linked articles or
downloads podcast media.

### Monitor multiple feeds

```json
{
  "feeds": [
    {"url": "https://example.com/feed.xml", "externalId": "competitor-blog"},
    {"url": "https://example.org/atom.xml", "externalId": "release-notes"},
    {"url": "https://example.net/feed.json", "externalId": "industry-news"}
  ],
  "mode": "monitor",
  "monitorId": "market-intelligence",
  "outputMode": "changesOnly",
  "maxItemsPerFeed": 100,
  "discoverFeedFromWebsite": false
}
```

RSS 2.0, Atom 1.0 and JSON Feed 1.1 share one output contract. Common RSS 1.0
(RDF) is accepted on a compatibility path. The Actor makes direct HTTP requests
by default; a proxy is optional and a residential proxy is not expected.

### Monitoring semantics

The first successful monitor run is a `BASELINE`. In `changesOnly` mode it emits
the `feed_summary`, not hundreds of historical item events:

```json
{
  "record_type": "feed_summary",
  "input_index": 0,
  "external_id": "competitor-blog",
  "status": "SUCCESS",
  "change_type": "BASELINE",
  "items_observed": 42,
  "baseline_items": 42,
  "new_items": 0,
  "updated_items": 0,
  "not_modified_304": false
}
```

On later successful runs:

- an unseen stable ID is `NEW_ITEM`;
- a known ID with a changed semantic fingerprint is `UPDATED_ITEM`;
- the same ID and fingerprint is `UNCHANGED`;
- an item absent from the current feed produces no event.

There is intentionally no `REMOVED_ITEM`: most feeds expose only their newest N
entries, so disappearance does not prove deletion.

A new item is immediately usable as an alert or ingestion payload:

```json
{
  "record_type": "feed_item",
  "source": "RSS / Atom / JSON Feed",
  "source_id": "feed-item:01b8...",
  "input_index": 0,
  "source_url": "https://example.com/posts/new-release",
  "scraped_at": "2026-09-01T12:00:00Z",
  "schema_version": 2,
  "fingerprint": "83d2...",
  "feed_url": "https://example.com/feed.xml",
  "feed_id": "https://example.com/feed.xml",
  "feed_title": "Example engineering blog",
  "external_id": "competitor-blog",
  "status": "SUCCESS",
  "change_type": "NEW_ITEM",
  "item_id": "post-1042",
  "item_id_source": "rss_guid",
  "guid": "post-1042",
  "url": "https://example.com/posts/new-release",
  "title": "New release",
  "summary": "What changed in this release.",
  "authors": [{"name": "Example Team"}],
  "published_at": "2026-09-01T10:30:00Z",
  "categories": ["Releases"],
  "raw_format": "rss"
}
```

Publishers can legitimately edit entries. Those rows include exact fields:

```json
{
  "record_type": "feed_item",
  "change_type": "UPDATED_ITEM",
  "item_id": "post-1042",
  "changed_fields": ["content_text", "title"]
}
```

HTTP metadata, feed position and scrape time never create an update.

### Conditional requests and 304

After each successful monitor check the Actor stores the canonical feed URL,
`ETag`, `Last-Modified`, last successful timestamp and a bounded item-ID map in
the named `rss-feed-monitor-state` KVS. Saved Actor Tasks are automatically
isolated by Task ID. Direct/API monitor runs must provide a stable `monitorId`;
there is no shared fallback scope that could mix independent workflows. The next
check replays available validators.
A proper `304` produces:

```json
{
  "record_type": "feed_summary",
  "status": "SUCCESS",
  "change_type": "UNCHANGED",
  "http_status": 304,
  "items_observed": 0,
  "etag_used": true,
  "last_modified_used": false,
  "not_modified_304": true
}
```

The previous item state remains intact and only the successful-check timestamp
advances. Feeds without validators still work: they are downloaded, parsed and
fingerprinted normally.

### Feed discovery

Set `discoverFeedFromWebsite` to `true` to inspect exactly one supplied homepage.
The Actor reads `<link rel="alternate">` declarations from its HTML head,
resolves relative URLs and selects RSS first, Atom second, JSON Feed third;
document order breaks ties. It does not guess common paths and does not crawl the
site. After the first success, scheduled runs use the stored canonical feed URL
directly.

### Podcast feeds

RSS enclosures are returned as `url`, `mime_type` and `length`. Straightforward
iTunes duration, episode and season values are attached to the enclosure. The
Actor never downloads audio or video files.

### Schedule polling and webhook integration

Create an Apify Schedule for this Actor with the monitor input above; polling
every 5–30 minutes is typical, but the publisher's update rate and terms should
decide the cadence. Keep the same Task (or the same `monitorId` for API runs) and
`externalId` so later runs diff against the same last successful state. Do not
overlap runs that share a monitoring scope; KVS does not provide a cross-run
compare-and-swap lock.

Add an Actor-run webhook for the `ACTOR.RUN.SUCCEEDED` event and point it at your
automation endpoint. The receiving workflow can read the run's default Dataset
and route only `NEW_ITEM` / `UPDATED_ITEM` rows. The accompanying
`feed_summary` rows show which feeds were checked, unchanged, blocked or failed,
even when there were zero item events.

For API-driven automation:

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/YOUR_USERNAME~rss-feed-monitor/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d @examples/sample_input.json
```

Store `$APIFY_TOKEN` as a secret; never paste a real token into source, Task
input or logs.

### Fresh-content AI/RAG workflow

Run the Actor on a schedule with `changesOnly`, send `content_text` (or
`summary` when the feed is summary-only) from `NEW_ITEM` and `UPDATED_ITEM` rows
to your chunk/embed step, and use `source_id` as the document key. On an update,
replace the vectors for that key. A 304 or an unchanged fingerprint does no RAG
work, which avoids duplicate embeddings.

### Use with AI agents through Apify MCP

Expose the Actor as a typed tool in an authenticated MCP-compatible client:

```text
https://mcp.apify.com?tools=automa-flow/rss-feed-monitor
```

Example prompt:

```text
Run automa-flow/rss-feed-monitor for these RSS, Atom, or JSON Feed URLs. Return
only NEW_ITEM and UPDATED_ITEM rows, keep each feed_summary status visible, and
use source_id as the downstream document key.
```

After publication the same URL becomes discoverable to Store users. A blocked
or failed feed never becomes a valid empty result and never overwrites its last
successful monitor state.

### Input

| Field | Default | Meaning |
| --- | --- | --- |
| `feeds` | required | 1–1,000 `{url, externalId?}` watchlist rows |
| `mode` | `snapshot` | `monitor` persists state; `snapshot` does not |
| `outputMode` | `all` | use `changesOnly` for NEW/UPDATED monitor events only |
| `maxItemsPerFeed` | `100` | current entries normalized per feed, maximum 500 |
| `discoverFeedFromWebsite` | `false` | inspect one homepage head for a declared feed |
| `monitorId` | Task ID or required | stable state namespace for direct recurring API runs |
| `proxyConfiguration` | disabled | optional Apify/custom proxy; direct HTTP is normal |

Each watchlist row is validated independently. One malformed URL does not stop
the other 999 rows.

### Output and failure semantics

Every input produces exactly one `feed_summary`. Item rows are additional.
`input_index` is the zero-based position of the feed in the submitted watchlist;
it also makes interrupted Dataset delivery unambiguous when an invalid duplicate
shares an `externalId` with another row.

| Status | Meaning |
| --- | --- |
| `SUCCESS` | HTTP and feed parsing completed; zero items may be a valid empty feed |
| `PARTIAL` | some feed entries conflicted or could not be represented; state is not advanced |
| `FAILED` | timeout, exhausted 429/5xx, DNS or unexpected operational failure |
| `BLOCKED` | the public source refused access with 401/403 |
| `INVALID_FEED` | malformed XML/JSON, HTML/login response, 404/410 or unsupported document |
| `INVALID_URL` | invalid/credential-bearing URL or SSRF protection rejected the target |
| `SKIPPED` | not fetched because the run's pay-per-event spending limit was reached |

`FAILED`, `PARTIAL`, malformed responses and exhausted retries never overwrite
good monitoring state. This is the difference between “the feed is empty” and
“the feed could not be verified.”

### Identity and retention

Identity priority is RSS `guid`, Atom entry `id`, JSON Feed item `id`, canonical
item URL, then a deterministic hash of stable semantic fields. The last fallback
cannot reliably recognize an entry whose title/date/authorship all change at
once; `item_id_source` makes that limitation visible.

State retains at most 2,000 IDs per feed and drops IDs not seen for 90 days.
That keeps KVS cost bounded. A very old pruned item that later reappears may be
classified as new; no unbounded history or external database is hidden behind
the Actor.

Very large text fields and nested arrays are bounded before Dataset delivery.
When an output value is shortened, `content_truncated_fields` names it; the
fingerprint and monitoring state still use the complete normalized feed value.

### Security and operating limits

- Only public HTTP(S) feeds are supported; login, CAPTCHA and browser sessions
  are deliberately excluded.
- Localhost, private/link-local/reserved/multicast IPs, cloud metadata targets,
  embedded credentials and redirects to them are blocked. Every DNS and
  redirect target is revalidated and the HTTP connection is pinned to the
  validated public IP while preserving the original Host header and TLS SNI.
- Compressed and decompressed feed bodies are independently capped at 10 MB;
  DNS, redirects, attempts and retry delays share a 30-second per-feed deadline.
  Retries and XML shape are also bounded. DTD/entity
  declarations are rejected; no external XML entity or network entity
  resolution is performed.
- Direct HTTP is normal. Optional datacenter proxy is user-selected; residential
  proxy and Playwright are not part of the design.
- Feed content remains subject to the publisher's terms and copyright. This
  Actor enables user-controlled extraction of public syndication data and does
  not claim additional permission or fetch linked articles.

### Pricing

The PPE price is **$0.004 per successful `feed-check`**, including a valid empty
feed or 304, plus **$0.00005 per delivered `feed-item` row**. Retries, invalid
input, FAILED, BLOCKED, PARTIAL and SKIPPED checks are not billed. In the
`monitor` + `changesOnly` workflow, BASELINE and unchanged entries produce no
item charge. If the run spending limit cannot cover a complete successful feed
result, that feed and all remaining inputs are emitted as `SKIPPED` without
advancing state. The standard `$0.00005` `apify-actor-start` event is enabled;
the automatic `apify-default-dataset-item` event must be disabled so summaries
and custom-charged item rows are never charged a second time. The Actor refuses
to run under that unsafe PPE configuration.

Before paid rows are delivered, each feed result is split into bounded batches
and staged temporarily in the run's default key-value store. If Apify migrates,
reboots or resurrects the same run, the Actor restores that exact plan, scans
the preserved Dataset identities and writes or charges only missing rows. A
successful summary is the completion marker; monitoring state is committed
before the recovery plan is removed. Final event counts are reconciled against
unique item `source_id` values and per-input summary identities; an overcharge
or unrecoverable mismatch fails the run instead of silently continuing.

The checked-in 512 MB 100/1,000-feed benchmark models 1,000 baseline checks at
about $0.2109 platform cost, an all-304 repeat at $0.1720, and 1,000 checks each
carrying one new item at $0.3170. These conservative totals include all
recovery-journal KVS reads, writes and cleanup. Customer PPE charges are
$4.00005, $4.00005 and $4.05005 respectively, including one Actor-start event.
The model uses a 195 KB/33-item feed matching the live-set averages, current
documented BRONZE rates and the 80% publisher revenue share. It also reports
publisher contribution and a conservative 10 MB/one-item tail estimate. Cloud
gates confirm 0.003885 CU for 100 independent inputs, 0.007005 CU for a complete
1,000-row failure-heavy batch, 0.027357 CU for 823 controlled successful checks
before the Free account cap, and 0.000233 CU for an accepted 9.10 MB/500-item
feed.

A 58.42-hour repeat over the same 200-host monitor emitted 1,793 unique change
rows — 1,737 new and 56 updated — with zero duplicate `source_id` or logical
item identities. It charged exactly 184 successful checks and 1,793 delivered
items; all non-successes remained uncharged and newly failed feeds retained
their last-good state. The event basket was $0.82570 with $0.023602 measured
resource usage, implying an estimated 77.14% paid-user contribution margin on
gross events. Its 255.74 MB peak under a 256 MiB test allocation led to the
safer 512 MB minimum/default. With recovery overhead, the refreshed offline
100/1,000-feed margins remain 72.17–75.70%. The 30-second simultaneous
10 MB/one-item ceiling is now positive at an estimated `$0.000206` contribution;
failed checks remain intentionally free and bounded. The price remains
justified. Same-run resurrection was cloud-verified on the 9.10 MB/500-item and
one-item fixtures: Dataset rows and custom event counts stayed one-to-one, no
feed was fetched again, and monitoring recovery completed. Apify charged a
second `$0.00005` synthetic start for the restarted container, as expected;
`feed-check` and `feed-item` were not charged again. The recovery journal was
empty after the final build completed.

# Actor input Schema

## `feeds` (type: `array`):

One to 1,000 public RSS, Atom or JSON Feed URLs. externalId is your optional stable watchlist ID and is echoed into output. With discovery enabled, a row may instead be a public website homepage declaring a feed in its HTML head.

## `mode` (type: `string`):

monitor stores the last successful feed state in KVS and classifies items. snapshot returns current items without reading or writing monitoring state.

## `outputMode` (type: `string`):

On a first monitor run, changesOnly emits only the BASELINE feed\_summary. Later it emits NEW\_ITEM and UPDATED\_ITEM rows plus every feed\_summary. Snapshot always returns current items.

## `maxItemsPerFeed` (type: `integer`):

Maximum entries normalized from each current feed response. This never follows article links or downloads podcast media.

## `discoverFeedFromWebsite` (type: `boolean`):

When enabled on the first run, inspect exactly one homepage HTML head and select a declared feed. Preference is RSS, then Atom, then JSON Feed, with document order breaking ties. No site crawl is performed.

## `monitorId` (type: `string`):

Stable scope for direct/API monitor runs that share the named state store. Saved Actor Tasks are isolated automatically by Task ID; otherwise monitorId is required. Keep this value unchanged across runs that should share history.

## `proxyConfiguration` (type: `object`):

Not required for normal operation. Direct HTTP is the measured default. If a public feed needs an Apify datacenter proxy, opt in here; residential proxy is not expected.

## Actor input object example

```json
{
  "feeds": [
    {
      "url": "https://news.ycombinator.com/rss",
      "externalId": "hacker-news"
    }
  ],
  "mode": "snapshot",
  "outputMode": "all",
  "maxItemsPerFeed": 100,
  "discoverFeedFromWebsite": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `runSummary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "feeds": [
        {
            "url": "https://news.ycombinator.com/rss",
            "externalId": "hacker-news"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("automa-flow/rss-feed-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "feeds": [{
            "url": "https://news.ycombinator.com/rss",
            "externalId": "hacker-news",
        }] }

# Run the Actor and wait for it to finish
run = client.actor("automa-flow/rss-feed-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "feeds": [
    {
      "url": "https://news.ycombinator.com/rss",
      "externalId": "hacker-news"
    }
  ]
}' |
apify call automa-flow/rss-feed-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automa-flow/rss-feed-monitor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/PSob9pjFHuCm0X1ik/builds/IkfbYPfAnSgsrfbBA/openapi.json