Sitemap Scraper

The Sitemap Scraper extracts every URL from any website's sitemap.xml — including nested sitemap indexes, gzipped sitemaps, and sitemaps discovered via robots.txt. It returns each URL with full metadata: lastmod, changefreq, priority, and the source sitemap file it came from.

No browser, no API key, no proxy required. The actor works with plain HTTP requests and handles 429 rate limits with automatic exponential backoff. It streams large sitemaps without loading them entirely into memory, so it stays fast and reliable even on sites with hundreds of thousands of URLs.

Why use this Sitemap Scraper?

• robots.txt discovery — automatically finds sitemaps declared in robots.txt, plus falls back to common paths like /sitemap.xml
• Recursive sitemap index handling — follows nested <sitemapindex> files to any depth, processing all child sitemaps
• Gzip support — transparently decompresses .xml.gz sitemaps without extra configuration
• Full metadata per URL — extracts lastmod, changefreq, priority, and sourceSitemap (which sitemap file each URL came from)
• Source sitemap tracking — every URL includes the exact sitemap file it was found in, essential for debugging and auditing large sites
• 429 retry with exponential backoff — automatically retries rate-limited requests up to 5 times with increasing delays
• Streaming XML parsing — uses a streaming parser for memory efficiency on large sitemaps
• URL deduplication — automatically deduplicates URLs found in multiple sitemaps
• Optional HTTP status check — enable HEAD requests to verify each URL returns a valid status code
• Change monitoring — track new, updated, and removed URLs across scheduled runs

In our 22 August 2026 comparison of the first 20 sitemap-related Apify Store results, the market leader apify/sitemap-extractor had a 17.7% failure rate (220/1,241 runs in 30 days). This actor targets zero failures through proper retry logic and error handling.

Pricing

$0.001 per run plus $0.0004 per URL extracted.

URLs extracted	Estimated cost
1	$0.0014
100	$0.041
1,000	$0.401
10,000	$4.001

The per-result model means small sites cost almost nothing, while large sites pay proportionally. No hidden proxy or compute surcharges.

Prices above cover the Actor's chargeable run and result events. Standard Apify charges for retaining or downloading data after a run may apply.

How to use this Actor

Provide one or more website base URLs. The actor automatically discovers sitemaps via robots.txt and common paths. You can also paste direct sitemap URLs.

Quick start — extract all URLs from a website

{
  "targets": ["https://apify.com"]
}

The prefilled example returns one URL to keep the first test fast and cheap. Set maxItemsPerTarget: 0 to get everything.

Direct sitemap URL

{
  "targets": ["https://example.com/sitemap.xml"]
}

Multiple targets with status checking

{
  "targets": ["https://apify.com", "https://example.com"],
  "includeStatusCheck": true,
  "maxItemsPerTarget": 500
}

Monitor new, updated, and removed records

Every successful run stores a snapshot in a named key-value store. Run the Actor on a schedule with the same monitorId and targets to get a change feed.

{
  "targets": ["https://example.com"],
  "monitorId": "daily-check",
  "onlyChangesSince": "2026-08-01T00:00:00.000Z"
}

The first run creates the baseline and labels every record new. Later runs compare against it:

changeType	Meaning
new	URL appeared for the first time
updated	URL metadata changed (e.g. lastmod changed); changedFields lists them
unchanged	URL and metadata are the same as last run
removed	URL disappeared from the sitemap

When onlyChangesSince is set, unchanged records are excluded. A removal is emitted once, on the run that first detects it.

API example

curl -X POST \
  "https://api.apify.com/v2/acts/daXenlo~sitemap-scraper/run-sync-get-dataset-items" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "targets": ["https://apify.com"],
    "maxItemsPerTarget": 10
  }'

For longer runs, start the Actor asynchronously and read its default dataset after the run succeeds.

Input options

Field	Type	Default	Description
targets	string[]	required	Website URLs or direct sitemap URLs
maxItemsPerTarget	integer	0 (all)	Limit URLs per target; 0 returns everything
maxDepth	integer	5	Max recursion depth for nested sitemap indexes
maxSitemaps	integer	100	Max sitemap files to process per target
timeoutMillis	integer	30000	HTTP request timeout in milliseconds
concurrency	integer	3	Number of concurrent HTTP requests
includeStatusCheck	boolean	false	Run HEAD requests for HTTP status codes
monitorId	string	"default"	Namespace for change tracking
onlyChangesSince	ISO timestamp	null	Filter to only return changes since this timestamp
useApifyProxy	boolean	false	Enable Apify Proxy (usually not needed)

Output

One dataset item per extracted URL:

{
  "target": "https://apify.com",
  "url": "https://apify.com/blog",
  "lastmod": "2026-08-15",
  "changefreq": "daily",
  "priority": 0.8,
  "sourceSitemap": "https://apify.com/sitemap.xml",
  "statusCode": null,
  "sourceUrl": "https://apify.com",
  "scrapedAt": "2026-08-22T10:00:00.000Z",
  "changeType": "new",
  "changeDetectedAt": "2026-08-22T10:00:00.000Z",
  "firstSeenAt": "2026-08-22T10:00:00.000Z",
  "lastSeenAt": "2026-08-22T10:00:00.000Z"
}

The run also writes a SUMMARY record with requested, successful, and failed target counts, the exported URL count, change totals, and completion time.

Use cases

• SEO audits — extract all indexable URLs a site exposes via sitemaps for coverage analysis
• Site migrations — generate a complete URL inventory to build redirect maps before cutover
• Content inventory — catalog all pages with lastmod dates for freshness analysis
• Broken link discovery — combine with HTTP status checks to find 404s in sitemaps
• Competitive research — map a competitor's content structure and publication frequency
• Crawl seed lists — produce clean, deduplicated URL lists for downstream scrapers or LLM pipelines

FAQ

How does it discover sitemaps?

The actor first checks robots.txt for Sitemap: directives. If none are found, it tries common paths like /sitemap.xml, /sitemap_index.xml, and /sitemap.xml.gz. You can also provide direct sitemap URLs in the targets field.

Do I need an account or API key for the target website?

No. Sitemaps are public XML files served over HTTP. The actor only reads publicly accessible sitemap endpoints.

Does it work with sitemap index files?

Yes. The actor recursively follows <sitemapindex> entries to any depth (configurable via maxDepth). Each nested sitemap is fetched and parsed, and all URLs are collected.

What about gzipped sitemaps?

Fully supported. The actor transparently decompresses .xml.gz files using Node's built-in zlib.

How does it handle rate limiting?

The actor automatically retries 429 responses up to 5 times with exponential backoff (1s, 2s, 4s, 8s, 16s). You can also reduce concurrency to be gentler on the target server.

Can I check if URLs in the sitemap are still live?

Yes. Enable includeStatusCheck to run HTTP HEAD requests on each URL. The statusCode field in the output tells you the HTTP response code.

Can this Actor detect removed URLs?

It reports a removal when a URL present in the previous snapshot is absent from the next successful run using the same monitorId.

Limitations and responsible use

• The actor only reads publicly accessible sitemap XML files. It does not crawl page content or bypass authentication.
• Sitemap data is provided by the website owner and may not reflect the actual site structure. Some URLs in sitemaps may return errors or redirects.
• The actor respects rate limits and uses backoff. It does not attempt to bypass anti-bot protections.
• Use this tool in compliance with the target website's terms of service.

Support

Report a problem or request a feature through the Issues tab on this Actor's page. Include the run ID and the input you used.