XML Sitemap URL Extractor & Checker

Extract a clean website URL inventory from sitemap.xml, sitemap indexes, gzipped sitemaps, robots.txt declarations, or a bare website root. Preserve sitemap metadata, filter URLs before delivery, and optionally check every page's live HTTP status.

Use it for SEO audits, website migrations, broken-link checks, crawl planning, content inventories, monitoring, datasets, and AI-agent workflows.

Common use cases

• Extract every URL from an XML sitemap or sitemap index
• Follow nested sitemap indexes and gzip files recursively
• Find broken, redirected, or non-HTML URLs before a migration
• Filter product, category, documentation, or blog URLs
• Build a complete URL inventory for SEO, analytics, or crawling
• Give an AI agent structured access to a website's published URL set

What the Actor returns

Each unique URL row can include:

• page URL and source sitemap
• sitemap nesting depth
• lastmod, changefreq, and priority metadata
• live HTTP status code when checking is enabled
• redirect destination and content type
• status-check error and extraction timestamp

Quick start

{
  "startUrls": ["https://apify.com/sitemap.xml"],
  "includeNested": true,
  "maxDepth": 5,
  "maxUrls": 1000,
  "includePatterns": [],
  "excludePatterns": ["?preview="],
  "lastModifiedAfter": "",
  "checkStatus": false,
  "statusConcurrency": 20
}

You can also submit a website root. The Actor checks robots.txt for declared sitemaps and falls back to /sitemap.xml when no declaration is available.

AI agent and API usage

This Actor is intentionally straightforward for tool-calling agents:

• startUrls is the only required input
• website roots, XML, gzip, and robots.txt inputs use the same interface
• maxUrls is a hard result and price ceiling
• filters run before rows are delivered
• status checking is explicit and disabled by default to keep first runs fast
• output has a stable, documented dataset view

Use the API, OpenAPI, MCP, Python, JavaScript, CLI, or integration examples in the Actor's API tab. A safe agent test uses one sitemap, maxUrls: 20, and checkStatus: false.

Example workflows

Audit a small set of sitemap pages for HTTP errors

{
  "startUrls": ["https://apify.com/sitemap.xml"],
  "includeNested": true,
  "maxUrls": 25,
  "checkStatus": true,
  "statusConcurrency": 10
}

Extract only selected URL paths

{
  "startUrls": ["https://example.com/sitemap.xml"],
  "includeNested": true,
  "maxUrls": 10000,
  "includePatterns": ["/products/", "/blog/"],
  "excludePatterns": ["?preview=", "/tag/"],
  "checkStatus": false
}

Pricing

Delivered URL rows cost $0.00008 each ($0.08 per 1,000) plus a $0.00005 Actor-start event and any platform usage shown by Apify before the run. Nested sitemap files and filtered-out entries are not dataset rows and do not create result charges.

Limits and responsible use

This Actor reads only the public sitemap and page endpoints supplied by the user. A sitemap is a discovery hint, not permission for every downstream use. Respect each site's terms, robots policies, rate limits, copyright, and applicable privacy law.