# Sitemap Reindex Trigger for RAG and Agent Freshness (`crawlware/sitemap-reindex-trigger`) Actor Tells your RAG pipeline or AI agent exactly which pages changed since the last check, so you re embed only what is new instead of the whole site. - **URL**: https://apify.com/crawlware/sitemap-reindex-trigger.md - **Developed by:** [Crawlware](https://apify.com/crawlware) (community) - **Categories:** AI, Developer tools - **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.00 / 1,000 actor starts This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Sitemap Reindex Trigger for RAG and Agent Freshness Tells your RAG pipeline or AI agent **exactly which pages changed** since the last check, so you re-crawl and re-embed only what's new instead of re-processing the whole site on every schedule. ### The problem Every RAG pipeline built on scraped or crawled website content has the same maintenance question: how do you keep the vector index fresh without re-scraping and re-embedding the entire site on every run? Most teams either over-crawl (expensive, slow) or under-crawl (stale answers). This actor is the small, cheap trigger step in between: point it at a site, and it tells you precisely which URLs are new, changed, or gone since last time, and nothing else re-runs unless something actually changed. ### Why not just diff the sitemap myself? Because sitemap `lastmod` dates are frequently missing or wrong. This isn't theoretical: checking apify's own official developer documentation site (docs.apify.com, 3,772 real URLs across 7 sitemap files) with this actor, **3,764 of those 3,772 URLs (99.8%) had no usable `lastmod` on either side** and couldn't be verified by metadata alone. A naive lastmod-only diff tool, which is what every other sitemap-diff actor on the Store is, would call that entire site "unchanged" almost indefinitely, even as the docs are actively updated. This actor optionally verifies content by hash for exactly the URLs metadata can't confirm, so it catches real changes other sitemap-diff tools structurally miss. ### What it does 1. Discovers a site's sitemap(s) via `robots.txt`, with conventional-path fallback if `robots.txt` is missing or silent. 2. Follows sitemap index files recursively (including gzipped sitemaps), up to configurable depth and file-count limits. 3. Compares the current URL list against the last saved snapshot for that domain, and classifies every URL as **added**, **removed**, **changed** (lastmod moved), or **unverifiable** (no usable lastmod on either side). 4. Optionally verifies "unverifiable" URLs by fetching and hashing their actual content, catching changes lastmod alone would miss. 5. Optionally POSTs a JSON summary to your own webhook the moment real changes are found, no separate Apify webhook configuration needed. 6. Runs as a Standby HTTP endpoint: call it synchronously, get a structured JSON answer back inline. Built for agents and scripts to call directly, not just for humans clicking "Start" in a console. ### Who it's for - **RAG pipeline builders** keeping a vector index in sync with a live site, without re-embedding everything on every run. - **AI agents** (via Apify's MCP server or x402 agentic payments) that need a small, structured "what changed" tool rather than a full crawl. - **Agencies and SEO/DevOps teams** who want sitemap-change alerts on their own schedule, without wiring up generic webhook infrastructure themselves. - **Anyone burned by a sitemap-diff tool that only trusts `lastmod`**, see above. ### How recurring / scheduled checks work This actor runs as an always-ready Standby endpoint, not a scheduled batch job. To check a site daily, weekly, or on whatever cadence you want, point your own scheduler, a cron job, a GitHub Action, a Zapier/Make scenario, or an agent's own task loop, at the endpoint on that schedule, once per site. That's a deliberate choice, not a limitation. It means you only pay for checks that actually run, rather than for a container that sits warm 24/7 waiting for the next scheduled run. See "Engineered for real usage economics" below, since this actually came out of a real cost bug we caught and fixed before launch. ### Use with AI agents via MCP / Standby Call it synchronously as: ```` GET https:///?url=\&verifyContentHash=true ```` and get the change summary back inline, no waiting on a queued Apify run. Small, structured JSON output, one or two required inputs, typically under thirty seconds to run for most sites. Example real response, checking blog.apify.com's 917-URL sitemap: ```json { "type": "summary", "domain": "blog.apify.com", "snapshotGroup": "default", "isFirstRun": true, "sitemapFiles": 5, "stats": { "totalUrls": 917, "added": 0, "removed": 0, "changed": 0, "unchanged": 0, "unverifiable": 0 }, "addedUrls": [], "removedUrls": [], "changedUrls": [], "unverifiableUrls": [], "timestamp": "2026-07-06T08:37:17.923Z" } ```` Example agent prompts once connected via Apify's MCP server: - "Check whether any pages changed on docs.example.com since last week and list what's new." - "Compare our staging and production sitemaps and tell me what's missing from staging." - "Set up a daily check on our docs site and only re-embed pages this tool flags as changed." ### Engineered for real usage economics Standby actors bill for however long their container stays warm, at memory allocated multiplied by time, regardless of whether it's actively processing or just idling between calls. Left at Apify's 4 GB default with no idle timeout, this would cost roughly $0.80 an hour just sitting idle, about $576 a month, for a niche where comparable actors show only zero to two monthly active users, meaning very light real-world traffic. We caught this ourselves during testing and fixed it before launch: this actor runs at 256 MB with a 60-second idle timeout, so it costs real money only in the brief window around actual use. In our own testing, a full check of docs.apify.com's entire 3,772-URL site, including 25 content-hash verification fetches, cost about $0.001 in platform compute. We're mentioning this because, as far as we can tell, no comparable actor documents its own cost profile this transparently, and if you're an agent economically deciding which tool to call, that should be checkable, not assumed. ### Input | Field | Type | Description | |---|---|---| | `url` | string | A domain, site root, or direct sitemap URL. Required for a Standby call. | | `snapshotGroup` | string | Track the same domain on independent schedules without state colliding. Default `"default"`. | | `verifyContentHash` | boolean | Double-check unverifiable URLs by content hash. Recommended for RAG-freshness use, given the `lastmod` gaps described above. Default `false`. | | `maxContentChecks` | integer | Caps extra fetches spent on content-hash verification. Default `25`. | | `webhookUrl` | string | POST a JSON summary here automatically when changes are found. | | `webhookOnEmptyChanges` | boolean | Call the webhook even with zero changes, as a heartbeat. Default `false`. | | `maxUrlsPerSitemap` | integer | Cap on URLs processed per site. Default `20000`. | ### Output reference | Field | Description | |---|---| | `isFirstRun` | `true` the first time a domain/snapshotGroup pair is checked. Baseline only, deltas start on the next call. | | `stats.totalUrls` | Total URLs found in the site's sitemap(s) this run. | | `stats.added` / `removed` / `changed` | Counts by change type, based on sitemap metadata. | | `stats.unverifiable` | URLs where neither run had a usable `lastmod`, candidates for content-hash verification. | | `stats.contentHashChecked` / `contentHashChanged` | Present only when `verifyContentHash` is on, how many unverifiable URLs were checked, and how many actually changed. | | `addedUrls` / `removedUrls` / `changedUrls` / `unverifiableUrls` / `contentHashChangedUrls` | The actual URL lists behind each count. | | `webhook` | Present if `webhookUrl` was set, whether delivery was attempted and whether it succeeded. | ### Pricing Pay-per-event: a small start fee, a per-1,000-URLs-processed fee, a per-diff fee, and a per-content-hash-check fee for the optional verification pass. You pay for what actually ran, not a flat subscription, and the free tier is capped so it costs us, not you, to try it. ### FAQ **How is this different from a plain sitemap URL extractor?** Extractors give you a URL list. This tracks state between calls and tells you what's different from last time, that's the part an automated re-index pipeline actually needs. **How is this different from other sitemap-diff actors?** Every other one we found trusts `lastmod` as ground truth and is written for human SEO/DevOps review, not agent calls. This one verifies by content hash where metadata can't be trusted, runs as a synchronous Standby endpoint built for agents, and is transparent about its own compute cost (see above). **Does this replace Apify's Schedule feature?** No, see "How recurring checks work" above. Point your own scheduler at the Standby URL instead. **What happens on the very first check of a new site?** It saves a baseline snapshot and reports zero added/removed/changed by design, there's nothing to compare against yet. Real deltas appear from the second check onward. **What if a site has no sitemap, or blocks robots.txt?** It fails gracefully for that domain and reports what it could find rather than erroring out the whole call. **Is this legal?** Yes. Sitemap files are public, intentionally published by site owners specifically so crawlers can discover content. This actor only reads public sitemap files (and, optionally, public page content for hash verification). It doesn't bypass authentication or access private data. # Actor input Schema ## `url` (type: `string`): A domain (example.com), a site root (https://example.com), or a direct sitemap URL. Recommended for single-site / agent calls. ## `snapshotGroup` (type: `string`): Lets you track the same domain on two independent schedules without their saved state colliding. Defaults to "default". ## `verifyContentHash` (type: `boolean`): Sitemap lastmod dates are frequently missing or unreliable. When on, pages the sitemap can't verify are double-checked by hashing their actual content, catching real changes lastmod-only tools miss. ## `maxContentChecks` (type: `integer`): Caps the extra fetches spent on content-hash verification, to keep cost predictable on large sites. ## `webhookUrl` (type: `string`): If set, a JSON summary is POSTed here automatically when changes are found -- no need to configure Apify's generic webhook feature separately. ## `webhookOnEmptyChanges` (type: `boolean`): Useful as a heartbeat/liveness signal for scheduled monitoring. ## `maxUrlsPerSitemap` (type: `integer`): Caps how many URLs are processed for very large sites. ## Actor input object example ```json { "url": "https://docs.apify.com", "verifyContentHash": false, "maxContentChecks": 25, "webhookOnEmptyChanges": false, "maxUrlsPerSitemap": 20000 } ``` # Actor output Schema ## `standbyEndpoint` (type: `string`): Call this URL with a url query parameter (and optional verifyContentHash, snapshotGroup, webhookUrl, etc.) to check a site for sitemap changes and get a JSON summary back inline. See the Standby tab for the full request and response schema. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "url": "https://docs.apify.com" }; // Run the Actor and wait for it to finish const run = await client.actor("crawlware/sitemap-reindex-trigger").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "url": "https://docs.apify.com" } # Run the Actor and wait for it to finish run = client.actor("crawlware/sitemap-reindex-trigger").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "url": "https://docs.apify.com" }' | apify call crawlware/sitemap-reindex-trigger --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=crawlware/sitemap-reindex-trigger", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Sitemap Reindex Trigger for RAG and Agent Freshness", "description": "Tells your RAG pipeline or AI agent exactly which pages changed since the last check, so you re embed only what is new instead of the whole site.", "version": "0.1", "x-build-id": "YOsxWe1lBrW1Y4C5M" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/crawlware~sitemap-reindex-trigger/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-crawlware-sitemap-reindex-trigger", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/crawlware~sitemap-reindex-trigger/runs": { "post": { "operationId": "runs-sync-crawlware-sitemap-reindex-trigger", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/crawlware~sitemap-reindex-trigger/run-sync": { "post": { "operationId": "run-sync-crawlware-sitemap-reindex-trigger", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "url": { "title": "Site or sitemap URL", "type": "string", "description": "A domain (example.com), a site root (https://example.com), or a direct sitemap URL. Recommended for single-site / agent calls." }, "snapshotGroup": { "title": "Snapshot group", "type": "string", "description": "Lets you track the same domain on two independent schedules without their saved state colliding. Defaults to \"default\"." }, "verifyContentHash": { "title": "Verify with content hash (recommended for RAG freshness)", "type": "boolean", "description": "Sitemap lastmod dates are frequently missing or unreliable. When on, pages the sitemap can't verify are double-checked by hashing their actual content, catching real changes lastmod-only tools miss.", "default": false }, "maxContentChecks": { "title": "Max content-hash checks per run", "type": "integer", "description": "Caps the extra fetches spent on content-hash verification, to keep cost predictable on large sites.", "default": 25 }, "webhookUrl": { "title": "Webhook URL", "type": "string", "description": "If set, a JSON summary is POSTed here automatically when changes are found -- no need to configure Apify's generic webhook feature separately." }, "webhookOnEmptyChanges": { "title": "Call webhook even with zero changes", "type": "boolean", "description": "Useful as a heartbeat/liveness signal for scheduled monitoring.", "default": false }, "maxUrlsPerSitemap": { "title": "Max URLs per site", "type": "integer", "description": "Caps how many URLs are processed for very large sites.", "default": 20000 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```