Sitemap Reindex Trigger for RAG and Agent Freshness

Tells your RAG pipeline or AI agent exactly which pages changed since the last check, so you re-crawl and re-embed only what's new instead of re-processing the whole site on every schedule.

The problem

Every RAG pipeline built on scraped or crawled website content has the same maintenance question: how do you keep the vector index fresh without re-scraping and re-embedding the entire site on every run? Most teams either over-crawl (expensive, slow) or under-crawl (stale answers). This actor is the small, cheap trigger step in between: point it at a site, and it tells you precisely which URLs are new, changed, or gone since last time, and nothing else re-runs unless something actually changed.

Why not just diff the sitemap myself?

Because sitemap lastmod dates are frequently missing or wrong. This isn't theoretical: checking apify's own official developer documentation site (docs.apify.com, 3,772 real URLs across 7 sitemap files) with this actor, 3,764 of those 3,772 URLs (99.8%) had no usable lastmod on either side and couldn't be verified by metadata alone. A naive lastmod-only diff tool, which is what every other sitemap-diff actor on the Store is, would call that entire site "unchanged" almost indefinitely, even as the docs are actively updated. This actor optionally verifies content by hash for exactly the URLs metadata can't confirm, so it catches real changes other sitemap-diff tools structurally miss.

What it does

1. Discovers a site's sitemap(s) via robots.txt, with conventional-path fallback if robots.txt is missing or silent.
2. Follows sitemap index files recursively (including gzipped sitemaps), up to configurable depth and file-count limits.
3. Compares the current URL list against the last saved snapshot for that domain, and classifies every URL as added, removed, changed (lastmod moved), or unverifiable (no usable lastmod on either side).
4. Optionally verifies "unverifiable" URLs by fetching and hashing their actual content, catching changes lastmod alone would miss.
5. Optionally POSTs a JSON summary to your own webhook the moment real changes are found, no separate Apify webhook configuration needed.
6. Runs as a Standby HTTP endpoint: call it synchronously, get a structured JSON answer back inline. Built for agents and scripts to call directly, not just for humans clicking "Start" in a console.

Who it's for

• RAG pipeline builders keeping a vector index in sync with a live site, without re-embedding everything on every run.
• AI agents (via Apify's MCP server or x402 agentic payments) that need a small, structured "what changed" tool rather than a full crawl.
• Agencies and SEO/DevOps teams who want sitemap-change alerts on their own schedule, without wiring up generic webhook infrastructure themselves.
• Anyone burned by a sitemap-diff tool that only trusts lastmod, see above.

How recurring / scheduled checks work

This actor runs as an always-ready Standby endpoint, not a scheduled batch job. To check a site daily, weekly, or on whatever cadence you want, point your own scheduler, a cron job, a GitHub Action, a Zapier/Make scenario, or an agent's own task loop, at the endpoint on that schedule, once per site. That's a deliberate choice, not a limitation. It means you only pay for checks that actually run, rather than for a container that sits warm 24/7 waiting for the next scheduled run. See "Engineered for real usage economics" below, since this actually came out of a real cost bug we caught and fixed before launch.

Use with AI agents via MCP / Standby

Call it synchronously as:

GET https://<your-standby-url>/?url=<site>&verifyContentHash=true

and get the change summary back inline, no waiting on a queued Apify run. Small, structured JSON output, one or two required inputs, typically under thirty seconds to run for most sites. Example real response, checking blog.apify.com's 917-URL sitemap:

{
  "type": "summary",
  "domain": "blog.apify.com",
  "snapshotGroup": "default",
  "isFirstRun": true,
  "sitemapFiles": 5,
  "stats": { "totalUrls": 917, "added": 0, "removed": 0, "changed": 0, "unchanged": 0, "unverifiable": 0 },
  "addedUrls": [], "removedUrls": [], "changedUrls": [], "unverifiableUrls": [],
  "timestamp": "2026-07-06T08:37:17.923Z"
}

Example agent prompts once connected via Apify's MCP server:

• "Check whether any pages changed on docs.example.com since last week and list what's new."
• "Compare our staging and production sitemaps and tell me what's missing from staging."
• "Set up a daily check on our docs site and only re-embed pages this tool flags as changed."

Engineered for real usage economics

Standby actors bill for however long their container stays warm, at memory allocated multiplied by time, regardless of whether it's actively processing or just idling between calls. Left at Apify's 4 GB default with no idle timeout, this would cost roughly $0.80 an hour just sitting idle, about $576 a month, for a niche where comparable actors show only zero to two monthly active users, meaning very light real-world traffic. We caught this ourselves during testing and fixed it before launch: this actor runs at 256 MB with a 60-second idle timeout, so it costs real money only in the brief window around actual use. In our own testing, a full check of docs.apify.com's entire 3,772-URL site, including 25 content-hash verification fetches, cost about $0.001 in platform compute. We're mentioning this because, as far as we can tell, no comparable actor documents its own cost profile this transparently, and if you're an agent economically deciding which tool to call, that should be checkable, not assumed.

Input

Field	Type	Description
url	string	A domain, site root, or direct sitemap URL. Required for a Standby call.
snapshotGroup	string	Track the same domain on independent schedules without state colliding. Default "default".
verifyContentHash	boolean	Double-check unverifiable URLs by content hash. Recommended for RAG-freshness use, given the lastmod gaps described above. Default false.
maxContentChecks	integer	Caps extra fetches spent on content-hash verification. Default 25.
webhookUrl	string	POST a JSON summary here automatically when changes are found.
webhookOnEmptyChanges	boolean	Call the webhook even with zero changes, as a heartbeat. Default false.
maxUrlsPerSitemap	integer	Cap on URLs processed per site. Default 20000.

Output reference

Field	Description
isFirstRun	true the first time a domain/snapshotGroup pair is checked. Baseline only, deltas start on the next call.
stats.totalUrls	Total URLs found in the site's sitemap(s) this run.
stats.added / removed / changed	Counts by change type, based on sitemap metadata.
stats.unverifiable	URLs where neither run had a usable lastmod, candidates for content-hash verification.
stats.contentHashChecked / contentHashChanged	Present only when verifyContentHash is on, how many unverifiable URLs were checked, and how many actually changed.
addedUrls / removedUrls / changedUrls / unverifiableUrls / contentHashChangedUrls	The actual URL lists behind each count.
webhook	Present if webhookUrl was set, whether delivery was attempted and whether it succeeded.

Pricing

Pay-per-event: a small start fee, a per-1,000-URLs-processed fee, a per-diff fee, and a per-content-hash-check fee for the optional verification pass. You pay for what actually ran, not a flat subscription, and the free tier is capped so it costs us, not you, to try it.

FAQ

How is this different from a plain sitemap URL extractor? Extractors give you a URL list. This tracks state between calls and tells you what's different from last time, that's the part an automated re-index pipeline actually needs.

How is this different from other sitemap-diff actors? Every other one we found trusts lastmod as ground truth and is written for human SEO/DevOps review, not agent calls. This one verifies by content hash where metadata can't be trusted, runs as a synchronous Standby endpoint built for agents, and is transparent about its own compute cost (see above).

Does this replace Apify's Schedule feature? No, see "How recurring checks work" above. Point your own scheduler at the Standby URL instead.

What happens on the very first check of a new site? It saves a baseline snapshot and reports zero added/removed/changed by design, there's nothing to compare against yet. Real deltas appear from the second check onward.

What if a site has no sitemap, or blocks robots.txt? It fails gracefully for that domain and reports what it could find rather than erroring out the whole call.

Is this legal? Yes. Sitemap files are public, intentionally published by site owners specifically so crawlers can discover content. This actor only reads public sitemap files (and, optionally, public page content for hash verification). It doesn't bypass authentication or access private data.