RSS Feed Scraper API & Incremental Monitor

Use this RSS feed scraper API to parse RSS, Atom and JSON Feed URLs and monitor them for genuinely new or updated items without processing duplicates on every run.

This Actor turns a feed watchlist into schedule- and webhook-ready events. It uses each feed's native item IDs, remembers only the last successful bounded state, and sends standard If-None-Match / If-Modified-Since validators on later runs. A publisher that answers 304 Not Modified costs one small check and no feed download or parse.

Quick start: RSS to JSON

{
  "feeds": [
    {"url": "https://news.ycombinator.com/rss", "externalId": "hacker-news"}
  ],
  "mode": "snapshot",
  "outputMode": "all",
  "maxItemsPerFeed": 20
}

Every current item becomes one normalized feed_item row. The Actor returns content already present in the feed; it never visits linked articles or downloads podcast media.

Monitor multiple feeds

{
  "feeds": [
    {"url": "https://example.com/feed.xml", "externalId": "competitor-blog"},
    {"url": "https://example.org/atom.xml", "externalId": "release-notes"},
    {"url": "https://example.net/feed.json", "externalId": "industry-news"}
  ],
  "mode": "monitor",
  "monitorId": "market-intelligence",
  "outputMode": "changesOnly",
  "maxItemsPerFeed": 100,
  "discoverFeedFromWebsite": false
}

RSS 2.0, Atom 1.0 and JSON Feed 1.1 share one output contract. Common RSS 1.0 (RDF) is accepted on a compatibility path. The Actor makes direct HTTP requests by default; a proxy is optional and a residential proxy is not expected.

Monitoring semantics

The first successful monitor run is a BASELINE. In changesOnly mode it emits the feed_summary, not hundreds of historical item events:

{
  "record_type": "feed_summary",
  "input_index": 0,
  "external_id": "competitor-blog",
  "status": "SUCCESS",
  "change_type": "BASELINE",
  "items_observed": 42,
  "baseline_items": 42,
  "new_items": 0,
  "updated_items": 0,
  "not_modified_304": false
}

On later successful runs:

• an unseen stable ID is NEW_ITEM;
• a known ID with a changed semantic fingerprint is UPDATED_ITEM;
• the same ID and fingerprint is UNCHANGED;
• an item absent from the current feed produces no event.

There is intentionally no REMOVED_ITEM: most feeds expose only their newest N entries, so disappearance does not prove deletion.

A new item is immediately usable as an alert or ingestion payload:

{
  "record_type": "feed_item",
  "source": "RSS / Atom / JSON Feed",
  "source_id": "feed-item:01b8...",
  "input_index": 0,
  "source_url": "https://example.com/posts/new-release",
  "scraped_at": "2026-09-01T12:00:00Z",
  "schema_version": 2,
  "fingerprint": "83d2...",
  "feed_url": "https://example.com/feed.xml",
  "feed_id": "https://example.com/feed.xml",
  "feed_title": "Example engineering blog",
  "external_id": "competitor-blog",
  "status": "SUCCESS",
  "change_type": "NEW_ITEM",
  "item_id": "post-1042",
  "item_id_source": "rss_guid",
  "guid": "post-1042",
  "url": "https://example.com/posts/new-release",
  "title": "New release",
  "summary": "What changed in this release.",
  "authors": [{"name": "Example Team"}],
  "published_at": "2026-09-01T10:30:00Z",
  "categories": ["Releases"],
  "raw_format": "rss"
}

Publishers can legitimately edit entries. Those rows include exact fields:

{
  "record_type": "feed_item",
  "change_type": "UPDATED_ITEM",
  "item_id": "post-1042",
  "changed_fields": ["content_text", "title"]
}

HTTP metadata, feed position and scrape time never create an update.

Conditional requests and 304

After each successful monitor check the Actor stores the canonical feed URL, ETag, Last-Modified, last successful timestamp and a bounded item-ID map in the named rss-feed-monitor-state KVS. Saved Actor Tasks are automatically isolated by Task ID. Direct/API monitor runs must provide a stable monitorId; there is no shared fallback scope that could mix independent workflows. The next check replays available validators. A proper 304 produces:

{
  "record_type": "feed_summary",
  "status": "SUCCESS",
  "change_type": "UNCHANGED",
  "http_status": 304,
  "items_observed": 0,
  "etag_used": true,
  "last_modified_used": false,
  "not_modified_304": true
}

The previous item state remains intact and only the successful-check timestamp advances. Feeds without validators still work: they are downloaded, parsed and fingerprinted normally.

Feed discovery

Set discoverFeedFromWebsite to true to inspect exactly one supplied homepage. The Actor reads <link rel="alternate"> declarations from its HTML head, resolves relative URLs and selects RSS first, Atom second, JSON Feed third; document order breaks ties. It does not guess common paths and does not crawl the site. After the first success, scheduled runs use the stored canonical feed URL directly.

Podcast feeds

RSS enclosures are returned as url, mime_type and length. Straightforward iTunes duration, episode and season values are attached to the enclosure. The Actor never downloads audio or video files.

Schedule polling and webhook integration

Create an Apify Schedule for this Actor with the monitor input above; polling every 5–30 minutes is typical, but the publisher's update rate and terms should decide the cadence. Keep the same Task (or the same monitorId for API runs) and externalId so later runs diff against the same last successful state. Do not overlap runs that share a monitoring scope; KVS does not provide a cross-run compare-and-swap lock.

Add an Actor-run webhook for the ACTOR.RUN.SUCCEEDED event and point it at your automation endpoint. The receiving workflow can read the run's default Dataset and route only NEW_ITEM / UPDATED_ITEM rows. The accompanying feed_summary rows show which feeds were checked, unchanged, blocked or failed, even when there were zero item events.

For API-driven automation:

curl -X POST \
  "https://api.apify.com/v2/acts/YOUR_USERNAME~rss-feed-monitor/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d @examples/sample_input.json

Store $APIFY_TOKEN as a secret; never paste a real token into source, Task input or logs.

Fresh-content AI/RAG workflow

Run the Actor on a schedule with changesOnly, send content_text (or summary when the feed is summary-only) from NEW_ITEM and UPDATED_ITEM rows to your chunk/embed step, and use source_id as the document key. On an update, replace the vectors for that key. A 304 or an unchanged fingerprint does no RAG work, which avoids duplicate embeddings.

Use with AI agents through Apify MCP

Expose the Actor as a typed tool in an authenticated MCP-compatible client:

https://mcp.apify.com?tools=automa-flow/rss-feed-monitor

Example prompt:

Run automa-flow/rss-feed-monitor for these RSS, Atom, or JSON Feed URLs. Return
only NEW_ITEM and UPDATED_ITEM rows, keep each feed_summary status visible, and
use source_id as the downstream document key.

After publication the same URL becomes discoverable to Store users. A blocked or failed feed never becomes a valid empty result and never overwrites its last successful monitor state.

Input

Field	Default	Meaning
feeds	required	1–1,000 {url, externalId?} watchlist rows
mode	snapshot	monitor persists state; snapshot does not
outputMode	all	use changesOnly for NEW/UPDATED monitor events only
maxItemsPerFeed	100	current entries normalized per feed, maximum 500
discoverFeedFromWebsite	false	inspect one homepage head for a declared feed
monitorId	Task ID or required	stable state namespace for direct recurring API runs
proxyConfiguration	disabled	optional Apify/custom proxy; direct HTTP is normal

Each watchlist row is validated independently. One malformed URL does not stop the other 999 rows.

Output and failure semantics

Every input produces exactly one feed_summary. Item rows are additional. input_index is the zero-based position of the feed in the submitted watchlist; it also makes interrupted Dataset delivery unambiguous when an invalid duplicate shares an externalId with another row.

Status	Meaning
SUCCESS	HTTP and feed parsing completed; zero items may be a valid empty feed
PARTIAL	some feed entries conflicted or could not be represented; state is not advanced
FAILED	timeout, exhausted 429/5xx, DNS or unexpected operational failure
BLOCKED	the public source refused access with 401/403
INVALID_FEED	malformed XML/JSON, HTML/login response, 404/410 or unsupported document
INVALID_URL	invalid/credential-bearing URL or SSRF protection rejected the target
SKIPPED	not fetched because the run's pay-per-event spending limit was reached

FAILED, PARTIAL, malformed responses and exhausted retries never overwrite good monitoring state. This is the difference between “the feed is empty” and “the feed could not be verified.”

Identity and retention

Identity priority is RSS guid, Atom entry id, JSON Feed item id, canonical item URL, then a deterministic hash of stable semantic fields. The last fallback cannot reliably recognize an entry whose title/date/authorship all change at once; item_id_source makes that limitation visible.

State retains at most 2,000 IDs per feed and drops IDs not seen for 90 days. That keeps KVS cost bounded. A very old pruned item that later reappears may be classified as new; no unbounded history or external database is hidden behind the Actor.

Very large text fields and nested arrays are bounded before Dataset delivery. When an output value is shortened, content_truncated_fields names it; the fingerprint and monitoring state still use the complete normalized feed value.

Security and operating limits

• Only public HTTP(S) feeds are supported; login, CAPTCHA and browser sessions are deliberately excluded.
• Localhost, private/link-local/reserved/multicast IPs, cloud metadata targets, embedded credentials and redirects to them are blocked. Every DNS and redirect target is revalidated and the HTTP connection is pinned to the validated public IP while preserving the original Host header and TLS SNI.
• Compressed and decompressed feed bodies are independently capped at 10 MB; DNS, redirects, attempts and retry delays share a 30-second per-feed deadline. Retries and XML shape are also bounded. DTD/entity declarations are rejected; no external XML entity or network entity resolution is performed.
• Direct HTTP is normal. Optional datacenter proxy is user-selected; residential proxy and Playwright are not part of the design.
• Feed content remains subject to the publisher's terms and copyright. This Actor enables user-controlled extraction of public syndication data and does not claim additional permission or fetch linked articles.

Pricing

The PPE price is $0.004 per successful feed-check, including a valid empty feed or 304, plus $0.00005 per delivered feed-item row. Retries, invalid input, FAILED, BLOCKED, PARTIAL and SKIPPED checks are not billed. In the monitor + changesOnly workflow, BASELINE and unchanged entries produce no item charge. If the run spending limit cannot cover a complete successful feed result, that feed and all remaining inputs are emitted as SKIPPED without advancing state. The standard $0.00005 apify-actor-start event is enabled; the automatic apify-default-dataset-item event must be disabled so summaries and custom-charged item rows are never charged a second time. The Actor refuses to run under that unsafe PPE configuration.

Before paid rows are delivered, each feed result is split into bounded batches and staged temporarily in the run's default key-value store. If Apify migrates, reboots or resurrects the same run, the Actor restores that exact plan, scans the preserved Dataset identities and writes or charges only missing rows. A successful summary is the completion marker; monitoring state is committed before the recovery plan is removed. Final event counts are reconciled against unique item source_id values and per-input summary identities; an overcharge or unrecoverable mismatch fails the run instead of silently continuing.

The checked-in 512 MB 100/1,000-feed benchmark models 1,000 baseline checks at about $0.2109 platform cost, an all-304 repeat at $0.1720, and 1,000 checks each carrying one new item at $0.3170. These conservative totals include all recovery-journal KVS reads, writes and cleanup. Customer PPE charges are $4.00005, $4.00005 and $4.05005 respectively, including one Actor-start event. The model uses a 195 KB/33-item feed matching the live-set averages, current documented BRONZE rates and the 80% publisher revenue share. It also reports publisher contribution and a conservative 10 MB/one-item tail estimate. Cloud gates confirm 0.003885 CU for 100 independent inputs, 0.007005 CU for a complete 1,000-row failure-heavy batch, 0.027357 CU for 823 controlled successful checks before the Free account cap, and 0.000233 CU for an accepted 9.10 MB/500-item feed.

A 58.42-hour repeat over the same 200-host monitor emitted 1,793 unique change rows — 1,737 new and 56 updated — with zero duplicate source_id or logical item identities. It charged exactly 184 successful checks and 1,793 delivered items; all non-successes remained uncharged and newly failed feeds retained their last-good state. The event basket was $0.82570 with $0.023602 measured resource usage, implying an estimated 77.14% paid-user contribution margin on gross events. Its 255.74 MB peak under a 256 MiB test allocation led to the safer 512 MB minimum/default. With recovery overhead, the refreshed offline 100/1,000-feed margins remain 72.17–75.70%. The 30-second simultaneous 10 MB/one-item ceiling is now positive at an estimated $0.000206 contribution; failed checks remain intentionally free and bounded. The price remains justified. Same-run resurrection was cloud-verified on the 9.10 MB/500-item and one-item fixtures: Dataset rows and custom event counts stayed one-to-one, no feed was fetched again, and monitoring recovery completed. Apify charged a second $0.00005 synthetic start for the restarted container, as expected; feed-check and feed-item were not charged again. The recovery journal was empty after the final build completed.