Sitemap URL Extractor - All URLs from robots.txt & sitemaps

Get every URL of a website in seconds, straight from its sitemaps, with last modified date, change frequency, priority and hreflang alternates. Type a domain: the Actor finds the sitemaps, follows the sitemap indexes, unzips .xml.gz files and returns one clean row per URL.

• Finds the sitemaps for you. It reads the Sitemap: lines of robots.txt, then tries /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml and /sitemap.txt. You can also give sitemap URLs directly.
• Handles real-world sitemaps. Nested sitemap indexes (up to 1,000 sitemaps per site), gzip files, CDATA, XML entities, text sitemaps (one URL per line) and files up to 60 MB.
• Never fails silently. When a site has no sitemap, or a sitemap returns 404, you get a row that says so. Error rows are free.
• Fast and cheap. Plain HTTP, no browser: 20,000 URLs from 56 sitemaps took 17 seconds in our test. $0.20 per 1,000 URLs.

What can I use it for?

• Crawl planning: get the full URL list of a site before scraping or auditing it, and filter to the sections you need.
• SEO audits: find pages with an old lastmod, check hreflang coverage, or compare the URLs in the sitemap with the pages Google indexed.
• Content monitoring: schedule a daily run with lastmodFrom to get only the new or updated pages of a competitor's blog.
• Link checking: send the URLs to Bulk URL Status Checker to find 404s and redirects.

Input

{
    "startUrls": ["wordpress.org", "https://example.com/post-sitemap.xml"],
    "maxUrls": 100000,
    "maxUrlsPerSite": 5000,
    "includePattern": "/news/",
    "excludePattern": "/tag/|/page/",
    "lastmodFrom": "2026-01-01"
}

A domain or a page URL is treated as a website, and its sitemaps are discovered from the root of the domain. Use includePattern to keep one section, for example /news/. With several websites, maxUrlsPerSite gives each one its share of maxUrls. URLs are deduplicated across all sitemaps and websites.

Output

One row per URL. Example from a real run:

{
    "url": "https://wordpress.org/about/privacy/cookies/",
    "lastmod": "2026-09-01T12:37:54Z",
    "changefreq": null,
    "priority": null,
    "sitemap": "https://wordpress.org/sitemap-1.xml",
    "site": "https://wordpress.org"
}

When a sitemap declares hreflang versions, the row also has "alternates": [{ "hreflang": "fr", "href": "..." }]. Error rows have site, sitemap and error only.

Pricing

$0.0002 per URL extracted ($0.20 per 1,000 URLs), plus Apify's small run-start fee. Error rows (no sitemap found, sitemap unreachable) are free. Use maxUrls to cap the cost of a run.

Fair use and safety

The Actor only downloads robots.txt and sitemap files, which are published for robots. It never visits the pages themselves. Private and internal network addresses are refused.

Use it from AI agents (MCP)

Add sitemap-url-extractor to the Apify MCP server in your MCP client (Claude, Cursor, VS Code...), so your agent can list the pages of any website before reading them.

Integrations

Apify API, schedules, webhooks, Make, Zapier, n8n. Export as JSON, CSV, Excel or HTML.

FAQ

The site has no sitemap. What can I do? Some sites (python.org, for example) don't publish one. In that case, use a crawler such as Apify's Website Content Crawler.

Why fewer URLs than the site really has? A sitemap only lists the pages its owner chose to declare. Pages missing from it are not discovered.

Something wrong or missing? Open an issue on the Issues tab.