# Feed Reindex Trigger — Multi-Feed RSS/Atom for RAG & Agents (`crawlware/feed-reindex-trigger`) Actor Multi-feed RSS/Atom monitor that dedupes by GUID and returns clean incremental Markdown since last run. Built for RAG pipelines and agents. - **URL**: https://apify.com/crawlware/feed-reindex-trigger.md - **Developed by:** [Crawlware](https://apify.com/crawlware) (community) - **Categories:** AI, Automation - **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.003 / item emitted This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Feed Reindex Trigger — Multi-Feed RSS/Atom Monitor for RAG & Agents Point it at as many RSS/Atom feeds as you track, and get back only the items that are actually new since the last check — deduped by GUID, returned as clean chunk-ready Markdown, not raw per-item JSON you have to post-process yourself. ### The problem Keeping a RAG index or an agent's context fresh from multiple content sources means polling a lot of feeds on a schedule. Most RSS actors solve half the problem: they'll parse one feed into JSON. They don't dedupe against what you already ingested, they don't batch many feeds behind one call, and they charge per item on every single poll -- including items you already have -- because they don't track state between runs at all. You end up building your own dedupe and diffing layer on top of someone else's parser, or paying for the same items again and again on every schedule tick. ### Why not just use another RSS actor? Because the Store's existing RSS actors fall into one of three buckets, and none of them solve the actual RAG/agent ingestion problem: - **Plain parsers** turn a feed into JSON and charge per item extracted -- every run, whether or not you've seen that item before. Point one at a fast-moving feed on an hourly schedule and you're re-buying the same headlines all week. - **AI-enrichment bolt-ons** add sentiment or summary scoring on top of RSS parsing, at 10-20x the per-item price of a plain parser -- useful if you want that specific analysis, wasteful if all you need is clean text to embed. - **Generic MCP wrappers** expose feed-fetching as a raw tool call with no opinion about incremental state, dedupe, or output shape -- which is exactly why one of these launched with textbook-perfect MCP framing and still has zero users five weeks later. Being MCP-callable isn't the differentiator; being useful to call is. This actor does the one thing none of them do: track what you've already seen, per feed, across runs, and only ever return what's actually new -- in the format a RAG pipeline wants to consume, not the format a parser happened to produce. ### What it does 1. Accepts a list of feed URLs (RSS 2.0, Atom, RSS 1.0/RDF) in a single call -- no separate actor run per feed. 2. Fetches and parses each feed, and compares every item's GUID (falling back to a link+title hash when a feed omits GUIDs) against a named, persistent snapshot for that feed -- not the default key-value store, which is purged between runs and would silently make every run look like the first one. 3. Returns only items that are new or materially changed since the last snapshot, as clean Markdown with front-matter metadata (source feed, title, link, published date, status), ready to chunk and embed. 4. Optionally filters by keyword before charging, so a broad feed can be narrowed to what actually matters to your pipeline. 5. Optionally posts a webhook the moment new items are found, for push-based ingestion instead of poll-and-check. 6. Runs as a Standby HTTP/MCP endpoint: call it synchronously and get structured Markdown back inline, built for agents and schedulers to call directly. One unreachable or malformed feed in a multi-feed call is reported inline as an error entry for that feed only -- it never fails the whole call. ### Who it's for - **RAG pipeline builders** monitoring many content sources (blogs, changelogs, news feeds, docs feeds) who need only what's new, in embeddable form, not a full re-parse every time. - **AI agents** (via Apify's MCP server / x402 agentic payments) that need a bounded, structured "what's new" tool rather than a raw feed dump. - **Newsletter and content-curation operators** who want deduped, clean source material across many feeds in one call instead of stitching together several single-feed actors. - **Anyone who's paid twice for the same RSS item** because their current tool has no memory between runs. ### How recurring / scheduled checks work Like Sitemap Reindex Trigger, this runs as an always-ready Standby endpoint rather than a scheduled batch job -- point your own scheduler (cron, GitHub Actions, Zapier/Make, or an agent's own task loop) at it on whatever cadence you want. Memory and idle-timeout are right-sized for a non-browser HTTP tool so the container doesn't sit warm (and billing you) between polls. ### Use with AI agents via MCP / Standby Call it synchronously with a list of feed URLs and get back only the new/updated items since your last call for each one, as Markdown blocks tagged with their source feed and metadata: ```` GET /?feedUrl=https://example.com/feed.xml GET /?feedUrls=https://a.com/feed.xml,https://b.com/rss\&snapshotGroup=myAgent\&keywords=launch,release ```` Example real response, first call against hnrss.org's front-page feed (baseline run -- see "What happens on the very first check" in the FAQ below): ```json { "type": "summary", "feedUrl": "https://hnrss.org/frontpage", "feedTitle": "Hacker News: Front Page", "snapshotGroup": "default", "isFirstRun": true, "stats": { "totalItems": 20, "newItems": 0, "updatedItems": 0, "filteredOut": 0 }, "newItems": [], "timestamp": "2026-07-07T06:38:32.221Z" } ```` Example agent prompts once connected via Apify's MCP server: - "Check our tracked blogs and changelogs for anything new since yesterday and summarize it." - "Monitor these five feeds and only tell me about posts mentioning 'pricing' or 'launch'." - "Set up a daily check on our competitor's changelog feed and post new entries to Slack." ### Engineered for real usage economics Like Sitemap Reindex Trigger, this actor runs at 256 MB with a 60-second idle timeout rather than Apify's 4 GB Standby default -- at default memory an always-on container would cost roughly $0.80/hour just for existing, regardless of traffic. Right-sized, a real Standby container lifecycle (spin up on request, handle it, idle out) measured at ~1 minute cost $0.001 in platform compute on a live test run. You pay for the brief window around actual use, not for a container sitting warm between polls. ### Input | Field | Description | |---|---| | `feedUrl` / `feedUrls` | Single feed (agent convenience) or a batch array, capped by `maxFeeds` (default 25). | | `snapshotGroup` | Track the same feed on two independent schedules without their state colliding. Defaults to `"default"`. | | `keywords` | Case-insensitive substring filter on title/content, applied before charging. | | `maxItemsPerFeed` | Cap on items read per feed per run (default 100). | | `maxFeeds` | Cap on feeds processed in one call (default 25). | | `webhookUrl` / `webhookOnEmptyChanges` | Optional push notification when new/updated items are found. | ### Output reference | Field | Description | |---|---| | `isFirstRun` | `true` the first time a feed/snapshotGroup pair is checked. Baseline only, seeds the snapshot -- real deltas start on the next call. | | `stats.totalItems` | Total items found in the feed this call. | | `stats.newItems` / `updatedItems` | Counts of genuinely new items (never seen before) and updated items (same GUID, changed content). | | `stats.filteredOut` | Items excluded by the `keywords` filter before charging. | | `newItems[].status` | `"new"` or `"updated"` for each returned item. | | `newItems[].markdown` | YAML front-matter (feed, title, link, publishedAt, author, categories, status) followed by the item body as clean, chunk-ready Markdown. | | `newItems[].dedupeKey` | The stable key used to track this item across calls -- the feed's GUID, or a fallback hash when a feed omits one. | | `webhook` | Present if `webhookUrl` was set: whether delivery was attempted and whether it succeeded. | | `type: "error"` entries | In a `feedUrls` batch call, a feed that failed to fetch or parse appears as its own `{ "type": "error", "input": "", "error": "..." }` entry rather than failing the whole call. | ### Pricing Charged per feed checked and per genuinely new/changed item returned -- never per item on a feed that hasn't moved since your last call. No reserved default events left over from the pricing wizard; every charge event matches something the code actually does (`run-start`, `feed-checked`, `item-emitted`). ### FAQ **How is this different from a plain RSS-to-JSON actor?** Plain parsers give you every item, every time, and leave dedupe and "what's new since last time" entirely to you. This tracks state between calls and only returns (and charges for) items that are actually new or changed. **How is this different from other feed monitors on the Store?** The incumbents we found are either plain per-item parsers with no memory between runs, AI-enrichment bolt-ons at several times the price for analysis most RAG pipelines don't need, or a generic MCP wrapper with no opinion on incremental state or output shape. This combines multi-feed batching, GUID dedupe against a persistent snapshot, and RAG-ready Markdown output in one call. **Does this replace Apify's Schedule feature?** No -- see "How recurring checks work" above. Point your own scheduler at the Standby URL instead of using Apify's Schedule feature. **What happens on the very first check of a new feed?** It saves a baseline snapshot and reports zero new/updated items by design -- there's nothing to compare against yet. Real deltas appear from the second check onward. **What if one feed in a `feedUrls` batch call is unreachable or malformed?** That feed's entry in the response is a `{ "type": "error", ... }` object with the error message; every other feed in the same call is still processed and returned normally. **Is this legal?** Yes. RSS and Atom feeds are public files intentionally published by site owners specifically so readers and tools can subscribe to them. This actor only reads public feed URLs you provide -- it doesn't bypass authentication or access private data. # Actor input Schema ## `feedUrl` (type: `string`): A single RSS, Atom, or RSS 1.0 (RDF) feed URL. Recommended for single-feed / agent calls. ## `feedUrls` (type: `array`): Many feed URLs checked in one call. Capped at maxFeeds per run to keep default runs fast and predictable. ## `snapshotGroup` (type: `string`): Lets you track the same feed on two independent schedules without their saved state colliding. Defaults to "default". ## `keywords` (type: `array`): If set, only items whose title or content contain at least one keyword (case-insensitive) are returned and charged for. ## `maxItemsPerFeed` (type: `integer`): Caps how many items are read from each feed per run, to keep cost and run time predictable on very large feeds. ## `maxFeeds` (type: `integer`): Caps how many feeds are processed in one call, to keep default runs comfortably under Apify's health-check window. ## `webhookUrl` (type: `string`): If set, a JSON summary is POSTed here automatically when new or updated items are found -- no need to configure Apify's generic webhook feature separately. ## `webhookOnEmptyChanges` (type: `boolean`): Useful as a heartbeat/liveness signal for scheduled monitoring. ## Actor input object example ```json { "feedUrl": "https://hnrss.org/frontpage", "maxItemsPerFeed": 100, "maxFeeds": 25, "webhookOnEmptyChanges": false } ``` # Actor output Schema ## `items` (type: `string`): Only items that are new or changed since the last check -- unchanged items are never returned or charged for. Each item includes status, feed source, title, link, publish date, and RAG-ready Markdown. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "feedUrl": "https://hnrss.org/frontpage" }; // Run the Actor and wait for it to finish const run = await client.actor("crawlware/feed-reindex-trigger").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "feedUrl": "https://hnrss.org/frontpage" } # Run the Actor and wait for it to finish run = client.actor("crawlware/feed-reindex-trigger").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "feedUrl": "https://hnrss.org/frontpage" }' | apify call crawlware/feed-reindex-trigger --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=crawlware/feed-reindex-trigger", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Feed Reindex Trigger — Multi-Feed RSS/Atom for RAG & Agents", "description": "Multi-feed RSS/Atom monitor that dedupes by GUID and returns clean incremental Markdown since last run. Built for RAG pipelines and agents.", "version": "0.1", "x-build-id": "gtS7uVii0OgMyHdfO" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/crawlware~feed-reindex-trigger/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-crawlware-feed-reindex-trigger", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/crawlware~feed-reindex-trigger/runs": { "post": { "operationId": "runs-sync-crawlware-feed-reindex-trigger", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/crawlware~feed-reindex-trigger/run-sync": { "post": { "operationId": "run-sync-crawlware-feed-reindex-trigger", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "feedUrl": { "title": "Feed URL", "type": "string", "description": "A single RSS, Atom, or RSS 1.0 (RDF) feed URL. Recommended for single-feed / agent calls." }, "feedUrls": { "title": "Feed URLs (batch)", "type": "array", "description": "Many feed URLs checked in one call. Capped at maxFeeds per run to keep default runs fast and predictable.", "items": { "type": "string" } }, "snapshotGroup": { "title": "Snapshot group", "type": "string", "description": "Lets you track the same feed on two independent schedules without their saved state colliding. Defaults to \"default\"." }, "keywords": { "title": "Keyword filter", "type": "array", "description": "If set, only items whose title or content contain at least one keyword (case-insensitive) are returned and charged for.", "items": { "type": "string" } }, "maxItemsPerFeed": { "title": "Max items per feed", "type": "integer", "description": "Caps how many items are read from each feed per run, to keep cost and run time predictable on very large feeds.", "default": 100 }, "maxFeeds": { "title": "Max feeds per call", "type": "integer", "description": "Caps how many feeds are processed in one call, to keep default runs comfortably under Apify's health-check window.", "default": 25 }, "webhookUrl": { "title": "Webhook URL", "type": "string", "description": "If set, a JSON summary is POSTed here automatically when new or updated items are found -- no need to configure Apify's generic webhook feature separately." }, "webhookOnEmptyChanges": { "title": "Call webhook even with zero new items", "type": "boolean", "description": "Useful as a heartbeat/liveness signal for scheduled monitoring.", "default": false } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```