Sitemap Extractor & Change Monitor: All URLs, lastmod, robots.txt, llms.txt (No Proxy)

Get every URL a website publishes in its sitemaps, as clean rows with lastmod, changefreq, priority and hreflang counts, plus one report per site. Point it at a domain and it finds the sitemaps for you. Run it in monitor mode on a schedule and it returns only the pages that were added, removed or updated since last time.

Built for:

• SEO teams and agencies. URL inventories before a migration or audit, index-bloat checks, and protocol errors in the sitemap itself.
• Competitor watching. A daily or weekly list of the pages a competitor just published or took down.
• RAG and AI-agent builders. The full URL list of a site before you crawl it, in one fast call.
• Developers. Sitemap-to-JSON without running a crawler, a browser or a proxy.

What it does

• Finds sitemaps automatically: reads Sitemap: lines in robots.txt, then tries common locations (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and others).
• Reads every format: sitemap indexes (nested), gzip (.xml.gz), plain-text sitemaps, and RSS 2.0 / Atom feeds.
• Returns one row per URL with lastmod, changefreq, priority, hreflang languages, image and video counts, and the sitemap file it came from. Duplicates across sitemaps are removed.
• Validates the sitemap protocol: invalid dates, future dates, bad changefreq/priority, non-HTTPS URLs, cross-host URLs, files over 50,000 URLs or 50 MB.
• Site report per domain: every sitemap file found (status, gzip, size, URL count, issues), totals, newest and oldest lastmod, robots.txt summary, and llms.txt / llms-full.txt presence.
• Monitor mode: saves a snapshot per site and, on the next run, outputs only added, removed and changed URLs, plus a CHANGES.csv.
• Optional status checks: HTTP status, final URL, content type and Last-Modified for each URL.

Quick start

1. Add a domain like https://www.example.com (or a sitemap URL) to Websites or sitemap URLs.
2. Click Start. Open the URLs tab for the URL list and Site reports for the summary.
3. To monitor changes: set Mode to Monitor changes since last run and add a schedule (daily or weekly). The first run saves the baseline; later runs return only what changed.

Sample output

A URL row:

{
  "recordType": "url",
  "site": "https://www.example.com",
  "url": "https://www.example.com/blog/launch",
  "sourceSitemap": "https://www.example.com/sitemap-blog.xml.gz",
  "sitemapKind": "urlset",
  "lastmod": "2026-10-20T10:00:00+00:00",
  "changefreq": "weekly",
  "priority": 0.8,
  "hreflangCount": 2,
  "hreflangLanguages": ["en", "es"],
  "imageCount": 1,
  "videoCount": 0,
  "issues": [],
  "changeType": "added",
  "previousLastmod": null
}

A site report (abbreviated):

{
  "recordType": "site",
  "site": "https://www.example.com",
  "discoveredVia": "robots",
  "robotsTxt": { "status": 200, "sitemapLines": ["https://www.example.com/sitemap_index.xml"] },
  "sitemaps": [{ "url": "https://www.example.com/sitemap-blog.xml.gz", "kind": "urlset", "status": 200, "gzip": true, "urlCount": 412, "issues": [] }],
  "totals": { "sitemapFiles": 4, "urlsUnique": 1873, "duplicates": 12, "truncated": false, "newestLastmod": "2026-10-23" },
  "llmsTxt": { "present": true, "h1": "Example", "sectionCount": 4, "llmsFullPresent": false },
  "monitor": { "firstRun": false, "added": 3, "removed": 1, "changed": 7, "unchanged": 1862 }
}

Pricing

Pay per event: you pay only for results.

Event	Free plan	Bronze	Silver	Gold
URL row output	$0.0006	$0.0005	$0.00045	$0.0004
Status check (only with "Check each URL's HTTP status")	$0.0006	$0.0005	$0.00045	$0.0004
Site report (once per site with a definitive answer)	$0.012	$0.01	$0.009	$0.008

That's $0.40–$0.60 per 1,000 URLs, plus one cent per site and Apify's standard Actor start fee. In monitor mode, unchanged URLs are free, so a daily check of a site with no changes costs about one cent. Not charged: excluded URLs, sites blocked by bot protection or unreachable, and status checks that time out or get blocked. Set Max cost per run to cap any run.

Integrations and API

Schedule it in Apify and send changes to Slack, email, Google Sheets, Make, Zapier or n8n through Apify's integrations, or call it from the Apify API. AI agents can use it through Apify's MCP server: give it a domain, get back the URL list.

Limits

• Reads sitemaps only. Pages that aren't in any sitemap or feed aren't listed (this is not a crawler).
• Monitor snapshots are kept in a storage named saiberops-sitemap-snapshots in your Apify account, so monitor runs must use the same account. Removed-URL detection is skipped when a run hits Max URLs per site.
• Sites behind bot protection (for example a Cloudflare challenge on the sitemap) are reported as POSSIBLY_BOT_BLOCKED and not charged. No proxies are used.

FAQ

Why no proxy? Sitemaps are published for machines to read. Plain, honest requests are enough for almost every site, and they keep runs fast and cheap.

What does SITEMAP_DISALLOWED_BUT_DECLARED mean? The site lists the sitemap in robots.txt (inviting crawlers to read it) but a Disallow rule also covers it. The Actor reads it because the site declared it, and flags the contradiction so you can fix it. Locations it only guesses at, and per-URL status checks, always obey Disallow.

My own site shows POSSIBLY_BOT_BLOCKED. Allowlist the SaiberOpsBot User-Agent in your firewall or CDN and re-run.

Does it read the content of llms.txt? Only its structure: presence, size, title (H1), number of sections and links, and whether an Optional section exists.

Responsible use

The Actor identifies itself as SaiberOpsBot, respects robots.txt by default (per RFC 9309), never logs in or solves CAPTCHAs, and does not rotate IP addresses. It outputs URLs, dates, counts and HTTP metadata only. Feed titles, descriptions and author names are never read into the output.

Support

Found a bug or need a feature? Open an issue on this Actor's Issues tab and we'll respond there.

Maintained by Saiber Operations LLC.