Sitemap Scraper - XML URL Extractor & Monitor

Sitemap Scraper - XML URL Extractor & Monitor helps you build a URL inventory from XML sitemaps for SEO audits, crawl planning or content monitoring. Discover sitemaps through robots.txt, follow nested indexes and gzip files, and export URLs with lastmod, changefreq, priority and their source sitemap.

Enter a website URL or a direct sitemap link. The Actor reads public XML without a target-site account; optional HTTP status checks help review listed URLs. Actor event charges at the default 256 MB memory are $0.001 per run plus $0.0004 per exported row: $0.401 for 1,000 rows.

Selected fields from a recent successful run; source values can change:

{
  "url": "https://apify.com/",
  "lastmod": null,
  "changefreq": null,
  "priority": null,
  "sourceSitemap": "https://apify.com/sitemap/pages.xml",
  "statusCode": null
}

Why use this Sitemap Scraper?

• robots.txt discovery — automatically finds sitemaps declared in robots.txt, plus falls back to common paths like /sitemap.xml
• Recursive sitemap index handling — follows nested <sitemapindex> files to any depth, processing all child sitemaps
• Gzip support — transparently decompresses .xml.gz sitemaps without extra configuration
• Full metadata per URL — extracts lastmod, changefreq, priority, and sourceSitemap (which sitemap file each URL came from)
• Source sitemap tracking — every URL includes the exact sitemap file it was found in, essential for debugging and auditing large sites
• 429 retry with exponential backoff — automatically retries rate-limited requests up to 5 times with increasing delays
• URL deduplication — automatically deduplicates URLs found in multiple sitemaps
• Optional HTTP status check — request each listed URL and report its HTTP status code
• Change monitoring — track new, updated, and removed URLs across scheduled runs

Temporary network failures are retried. Availability and coverage depend on the source website and the crawl limits you choose.

Pricing

$0.001 per run plus $0.0004 per URL extracted.

URLs extracted	Estimated cost
1	$0.0014
100	$0.041
1,000	$0.401
10,000	$4.001

The per-result model means small sites cost almost nothing, while large sites pay proportionally. No hidden proxy or compute surcharges.

Prices above cover the Actor's chargeable run and result events. Standard Apify charges for retaining or downloading data after a run may apply.

How to use this Actor

Provide one or more website base URLs. The actor automatically discovers sitemaps via robots.txt and common paths. You can also paste direct sitemap URLs.

Quick start: inspect one sitemap URL

{
  "targets": [
    "https://apify.com"
  ],
  "maxItemsPerTarget": 1
}

Click Start, inspect the Dataset, then export JSON or CSV. Inspect url and sourceSitemap before increasing the output limit. Metadata can be null when the publisher omits it; a sitemap entry does not prove that a page is currently reachable.

The prefilled example returns one URL to keep the first test fast and cheap. Set maxItemsPerTarget: 0 to get everything.

Direct sitemap URL

{
  "targets": ["https://example.com/sitemap.xml"]
}

Multiple targets with status checking

{
  "targets": ["https://apify.com", "https://example.com"],
  "includeStatusCheck": true,
  "maxItemsPerTarget": 500
}

Monitor new, updated, and removed records

Use at most 39 characters in monitorId to keep independent schedules distinct. Longer values are shortened for the monitor snapshot.

Every successful run stores a snapshot in a named key-value store. Run the Actor on a schedule with the same monitorId and targets to get a change feed.

{
  "targets": ["https://example.com"],
  "monitorId": "daily-check",
  "onlyChangesSince": "2026-08-01T00:00:00.000Z"
}

The first run creates the baseline and labels every record new. Later runs compare against it:

changeType	Meaning
new	URL appeared for the first time
updated	URL metadata changed (e.g. lastmod changed); changedFields lists them
unchanged	URL and metadata are the same as last run
removed	URL disappeared from the sitemap

When onlyChangesSince is set, unchanged records are excluded. A removal is emitted once, on the run that first detects it.

API example

curl -X POST \
  "https://api.apify.com/v2/acts/Clintsa~sitemap-scraper/run-sync-get-dataset-items" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "targets": ["https://apify.com"],
    "maxItemsPerTarget": 10
  }'

For longer runs, start the Actor asynchronously and read its default dataset after the run succeeds.

Input options

Field	Type	Default	Description
targets	string[]	required	Website URLs or direct sitemap URLs
maxItemsPerTarget	integer	0 (all)	Limit URLs per target; 0 returns everything
maxDepth	integer	5	Max recursion depth for nested sitemap indexes
maxSitemaps	integer	100	Max sitemap files to process per target
timeoutMillis	integer	30000	HTTP request timeout in milliseconds
concurrency	integer	3	Number of concurrent HTTP requests
includeStatusCheck	boolean	false	Run HTTP requests for HTTP status codes
monitorId	string	"default"	Namespace for change tracking
onlyChangesSince	ISO timestamp	null	Filter to only return changes since this timestamp
useApifyProxy	boolean	false	Enable Apify Proxy (usually not needed)

Output

One dataset item per extracted URL:

{
  "target": "https://apify.com",
  "url": "https://apify.com/blog",
  "lastmod": "2026-08-15",
  "changefreq": "daily",
  "priority": 0.8,
  "sourceSitemap": "https://apify.com/sitemap.xml",
  "statusCode": null,
  "sourceUrl": "https://apify.com",
  "scrapedAt": "2026-08-22T10:00:00.000Z",
  "changeType": "new",
  "changeDetectedAt": "2026-08-22T10:00:00.000Z",
  "firstSeenAt": "2026-08-22T10:00:00.000Z",
  "lastSeenAt": "2026-08-22T10:00:00.000Z"
}

The run also writes a SUMMARY record with requested, successful, and failed target counts, the exported URL count, change totals, and completion time.

Use cases

• SEO audits — extract all indexable URLs a site exposes via sitemaps for coverage analysis
• Site migrations — generate a complete URL inventory to build redirect maps before cutover
• Content inventory — catalog all pages with lastmod dates for freshness analysis
• Broken link discovery — combine with HTTP status checks to find 404s in sitemaps
• Competitive research — map a competitor's content structure and publication frequency
• Crawl seed lists — produce clean, deduplicated URL lists for downstream scrapers or LLM pipelines

FAQ

How does it discover sitemaps?

The actor first checks robots.txt for Sitemap: directives. If none are found, it tries common paths like /sitemap.xml, /sitemap_index.xml, and /sitemap.xml.gz. You can also provide direct sitemap URLs in the targets field.

Do I need an account or API key for the target website?

No. Sitemaps are public XML files served over HTTP. The actor only reads publicly accessible sitemap endpoints.

Does it work with sitemap index files?

Yes. The actor recursively follows <sitemapindex> entries to any depth (configurable via maxDepth). Each nested sitemap is fetched and parsed, and all URLs are collected.

What about gzipped sitemaps?

Fully supported. The actor transparently decompresses .xml.gz files using Node's built-in zlib.

How does it handle rate limiting?

The actor automatically retries 429 responses up to 5 times with exponential backoff (1s, 2s, 4s, 8s, 16s). You can also reduce concurrency to be gentler on the target server.

Can I check if URLs in the sitemap are still live?

Yes. Enable includeStatusCheck to run HTTP HTTP requests on each URL. The statusCode field in the output tells you the HTTP response code.

Can this Actor detect removed URLs?

It reports a removal when a URL present in the previous snapshot is absent from the next successful run using the same monitorId.

Limitations and responsible use

• The actor only reads publicly accessible sitemap XML files. It does not crawl page content or bypass authentication.
• Sitemap data is provided by the website owner and may not reflect the actual site structure. Some URLs in sitemaps may return errors or redirects.
• The actor respects rate limits and uses backoff. It does not attempt to bypass anti-bot protections.
• Use this tool in compliance with the target website's terms of service.

Support

Report a problem or request a feature through the Issues tab on this Actor's page. Include the run ID and the input you used.

Related tools: OpenCart Catalog Scraper - Products & Images.