Sitemap URL Extractor: All Pages, New & Removed URLs

Urlwright — Little tools for the web.

Plain-HTTP Apify Actor that discovers sitemaps, extracts URLs (indexes, nested sitemaps, .gz), optionally crawls the same host (or optional subdomains) when sitemaps are missing/empty, optionally diffs against the previous run, and optionally checks HTTP status. One bad domain never fails the whole run; a RUN_SUMMARY is written to the key-value store.

Default User-Agent: UrlwrightBot/1.0 (sitemap; +https://urlwright.com/bot) (UrlwrightBot for robots.txt matching).

What it does

1. Accept domains, site origins, or direct sitemap URLs.
2. Discover sitemaps via robots.txt Sitemap: lines, then common paths if needed.
3. Parse urlsets and sitemap indexes (bounded depth), including gzip.
4. Optional crawl fallback (crawlFallback: off | whenNoSitemap | always): BFS for href + rel=canonical only — no page body stored, no personal data. Emits discovered in-scope links and each successfully crawled HTML page (including the start page) as source: crawl rows. Always honors robots.txt + crawl-delay per host (even if respectRobotsTxt is false for sitemaps). Optional includeSubdomains allows the same registrable domain (Public Suffix List, including private suffixes such as github.io / vercel.app). Crawl URL rows are flushed in batches during the crawl when change detection is off.
5. Optional change detection (compareWithPrevious): named key-value store per domain, changeStatus = new | removed | unchanged. Use with Apify Schedules; set onlyChanges to emit only new/removed (cheaper). Subdomain crawl URLs stay under the start domain’s snapshot.
6. Optionally HEAD/GET each URL for status, redirects, and final URL — in blocks of 100: check a block, push it, charge for that block, then continue (so a timeout mid-run still yields rows).
7. Dataset rows are written in batches of 500 (trimmed to remaining pay-per-event / maxUrls budget).

Input example

Store console prefill uses a small sample (startUrls: https://docs.apify.com, maxUrls: 100). The maxUrls default for real runs remains 10000.

{
  "startUrls": ["https://docs.apify.com", "https://other.example/sitemap.xml"],
  "checkStatus": true,
  "maxUrls": 5000,
  "maxSitemapDepth": 5,
  "crawlFallback": "whenNoSitemap",
  "includeSubdomains": false,
  "maxCrawlPages": 100,
  "maxCrawlDepth": 2,
  "compareWithPrevious": true,
  "onlyChanges": true,
  "concurrency": 5,
  "requestTimeoutSecs": 20,
  "requestsPerSecond": 2,
  "respectRobotsTxt": true
}

Output

Output example (dataset item)

{
  "url": "https://example.com/about",
  "lastmod": "2026-02-01",
  "changefreq": "monthly",
  "priority": "0.5",
  "sourceSitemap": "https://example.com/sitemap.xml",
  "source": "sitemap",
  "changeStatus": "new",
  "statusCode": 200,
  "finalUrl": "https://example.com/about",
  "redirectChain": ["https://example.com/about-old"]
}

source is sitemap or crawl.

Run summary

Key-value store record RUN_SUMMARY contains per-domain errors, crawl/change counts, totals, and truncation metadata:

• truncated — true when the run stopped early and did not finish the full inventory.
• stoppedReason (optional) — why it stopped:
  • charge-limit — pay-per-event budget / maxTotalChargeUsd exhausted.
  • max-urls — maxUrls cap reached.
  • timeout — cooperative stop shortly before the Actor run deadline so RUN_SUMMARY can still be written. A domain stopped for this reason is marked truncated (not skippedReason: 'no-results-timeout', which is the separate 60s empty-domain rule).
• Per-domain truncated: true when that domain was incompletely written.
• changes.new / changes.removed / changes.unchanged — counts of dataset rows actually written with that changeStatus.
• changes.inventory.{new,removed,unchanged} — diff over the found inventory, capped by maxUrls when the domain was truncated (same inventory budget as on non-compare runs; large sites are not loaded in full just for compare). With onlyChanges, written unchanged is 0 while inventory.unchanged can be greater than 0.
• partial / removedCandidates — still set on incomplete compare runs (removedCandidates = inventory.removed); snapshot is not updated and removed rows are not emitted until the domain is complete.

Named snapshot stores: change-detection snapshots are written to separate named key-value stores (c04-url-snapshot-<hostname>-<hash>), not to the default store. The default store holds RUN_SUMMARY.

Input dedup: startUrls entries that resolve to the same target are processed once (first wins). Duplicate domain/origin inputs (same normalized origin) and duplicate direct sitemap URLs (same normalized URL) are skipped — no fetches, no charges — with a warning and skippedDuplicate: true in RUN_SUMMARY. A domain input and a direct sitemap URL for the same host are not duplicates of each other.

Scheduled change monitoring

1. Enable compareWithPrevious (and usually onlyChanges).
2. Create an Apify Schedule for this Actor / task.
3. Each run updates a named store and tags URLs with changeStatus. Store naming (Apify-safe, ≤63 chars):
   • Domain/origin inputs — c04-url-snapshot-<hostname>-<hash-of-host> (unchanged; existing scheduled monitors keep their store).
   • Direct sitemap URL inputs — same host slug, but the trailing hash is of the normalized sitemap URL, so two sitemap URLs on the same host get separate stores and do not overwrite each other’s snapshots. Per run, each snapshot store is read/written at most once (at most one domain-compared charge per store). Snapshots are written only when the domain inventory is complete; incomplete runs set changes.partial, keep written changes.removed at the rows actually emitted (usually 0; candidate count in removedCandidates / inventory.removed), and leave the previous snapshot unchanged.

Pricing

Pay per event (USD). Recommended minimum Max total charge per run: $0.02.

Event	When	FREE	BRONZE	SILVER	GOLD
url-extracted (primary)	Each dataset row, including synthetic changeStatus: removed rows	0.0003	0.00025	0.0002	0.00015
url-status-checked	After save, only when a real HTTP status was obtained (incl. 4xx/5xx)	0.0003	0.00025	0.0002	0.00015
crawled-page	Each successfully fetched HTML page during crawl fallback	0.0025	0.002	0.0017	0.0015
domain-compared	Once per domain after a complete snapshot update when a previous snapshot existed (not the first run; also when onlyChanges finds no diffs)	0.002	0.002	0.002	0.002
domain-empty	Once per input domain that yielded no URL rows (unreachable, no sitemap, blocked by robots.txt, crawl-delay skip, no-results timeout, empty sitemaps, or crawl that produced zero rows — e.g. non-HTML / disallowed start page with no other URLs)	0.0005	0.0005	0.0005	0.0005
apify-actor-start	Once per run	0.0002	0.0002	0.0002	0.0002

Platinum and Diamond pay the Gold price.

Bronze examples: $0.25 / 1,000 URLs · $0.25 / 1,000 status checks · $2 / 1,000 crawled pages · $0.002 per monitoring compare · $0.0005 per empty domain.

Limitations

• No browser / JavaScript-rendered sitemaps or SPAs.
• No personal data fields — only URLs and HTTP metadata (crawl stores no HTML body).
• respectRobotsTxt (default on) controls sitemap discovery, sitemap extraction, and status checks. Crawl always respects robots.txt (and crawl-delay) per host, including redirect hops and each subdomain’s own robots.txt.
• Crawl-delay cap: if robots.txt sets a crawl-delay above 10 seconds for our User-Agent, the domain is skipped (no sitemap/status/crawl requests beyond robots.txt; skippedReason: 'crawl-delay' in RUN_SUMMARY). Subdomains with an excessive crawl-delay are not crawled when includeSubdomains is on.
• No-result timeout: if a domain yields no URLs within 60 seconds, processing stops (skippedReason: 'no-results-timeout').
• requestsPerSecond minimum is 0.5 (default 2). Default memory 256 MB; max memory 512 MB.
• Private-network block: if the start URL itself is rejected (hostname resolves to a private or special-use address, an IP literal, an IPv4-mapped IPv6 address, or a redirect onto one of those), the domain is ssrf_blocked in RUN_SUMMARY (errorCode: SSRF_BLOCKED). It is not reported as a robots.txt failure, “No sitemap found”, or domain-empty, and no event is charged. A nested sitemap or a later crawl page that is blocked does not change the rest of the domain; only that blocked request is skipped and not charged.
• robots.txt fetch failures (RFC 9309): network errors and HTTP 5xx when fetching robots.txt are treated as disallow-all (for crawl and, when respectRobotsTxt is on, for sitemap discovery/extraction/status). HTTP 4xx on robots.txt is allow-all. Such failures mark the domain inventory incomplete (changes.partial) so snapshots are not overwritten with false “everything removed” diffs. A normal empty discovery (“No sitemap found”) is separate from these reliability failures. A start URL blocked as ssrf_blocked is not this case.
• Crawl stays on the same hostname by default; with includeSubdomains: true it may follow subdomains of the same registrable domain (never foreign domains; private multi-tenant suffixes like *.github.io / *.vercel.app are separate registrable domains). HTML download capped at 2 MB per page. Link extraction ignores <script> / <style> contents and non-http(s) schemes (javascript:, data:, mailto:, tel:, vbscript:).
• Content-Type: crawl downloads a page body only when Content-Type is text/html or application/xhtml+xml. Missing Content-Type is treated as non-HTML (body not downloaded, page not charged as crawled-page).
• Before a crawl, part of the remaining crawl pay-per-event budget is reserved for later URL extraction charges. This reserve applies only to the pay-per-event budget, never to maxCrawlPages.
• Sitemap download/decompress capped at 50 MB per file.
• Very large sites: use maxUrls / maxSitemapDepth / maxCrawlPages / maxCrawlDepth to bound cost and runtime.
• Status checks are polite (per-host rate limit + crawl-delay) and run in blocks of 100 interleaved with dataset pushes.

FAQ

Why is my dataset empty?

Common causes: the domain was unreachable or had no usable sitemap; crawl fallback was off or produced no in-scope HTML links; robots.txt / crawl-delay blocked the host; the 60s no-result timeout fired; or onlyChanges found nothing new or removed. Check RUN_SUMMARY for per-domain errors, skippedReason, and domain-empty charges.

What does a run with 1,000 URLs cost?

On Bronze, start + 1,000 extracted URLs (no status checks, no crawl): 0.0002 + 1000 × 0.00025 = $0.2502. Status checks and crawled pages are charged separately if enabled.

Does the Actor respect robots.txt?

Yes for crawl — always, including crawl-delay, regardless of respectRobotsTxt. For sitemap discovery, sitemap extraction, and status checks, respectRobotsTxt defaults to on (you can turn it off for those paths only).

How does change detection work?

Enable compareWithPrevious (and usually onlyChanges). Each complete run stores a URL snapshot in a named key-value store and tags dataset rows with changeStatus (new / removed / unchanged). Schedule the Actor so successive runs compare against the previous snapshot. Incomplete runs do not overwrite the previous snapshot.