Sitemap URL Extractor

Get the full list of URLs a website publishes in its XML sitemaps, with each URL's lastmod, changefreq and priority, in one dataset you can export to CSV, Excel or JSON.

Give it a domain and it finds the sitemaps itself: it reads the Sitemap: lines in robots.txt, or tries /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml and /wp-sitemap.xml. It follows sitemap index files to every child sitemap, reads gzip-compressed .xml.gz sitemaps and plain-text sitemaps, and removes duplicate URLs.

It fetches only robots.txt and sitemap files, never the pages themselves, so it is fast and light on the website: a site with 20,000 URLs typically takes well under a minute.

Use it for

• SEO audits and migrations: the complete list of URLs a site wants indexed, before and after a move.
• Content inventories: every blog post or product page, with last-modified dates.
• Change tracking: run it on a schedule with "Only URLs modified after" to see what changed recently.
• Feeding other tools: a clean URL list to pass to a crawler, a link checker or an AI pipeline.

Input

Field	What it does
Websites or sitemap URLs (required)	example.com, https://example.com, or a sitemap URL such as https://example.com/post-sitemap.xml (also .xml.gz, .txt).
Keep URLs matching	Regular expressions (case-insensitive); keep a URL if any matches, e.g. /blog/.
Drop URLs matching	Regular expressions; drop a URL if any matches, e.g. /tag/.
Only URLs modified after	A date. URLs without a lastmod are dropped when this is set.
Max URLs per site	Stop after this many URLs for each site. 0 = no limit.
Only URLs on the same host	Drop sitemap entries pointing to other hosts.

Output

{
  "url": "https://wordpress.org/news/2014/04/smith/",
  "lastmod": "2014-04-15T21:49:40Z",
  "changefreq": null,
  "priority": null,
  "sitemapUrl": "https://wordpress.org/news/video-sitemap-1.xml",
  "site": "https://wordpress.org",
  "scrapedAt": "2026-11-15T04:20:26.365Z"
}

lastmod, changefreq and priority are exactly as the site wrote them (or null). A RUN_SUMMARY record in the run's key-value store lists, per site, the sitemap files read, URLs listed and returned, sitemaps that failed (with the HTTP status or reason) and any that robots.txt disallows.

Pricing

Pay per result: $0.0005 per URL returned ($0.50 per 1,000 URLs), plus Apify's small run-start fee. URLs removed by your filters or as duplicates are not charged. Large sites can have millions of URLs: use the filters, "Max URLs per site", or a maximum cost per run, which the Actor respects and stops at cleanly.

Limits and honest notes

• It lists what the sitemaps say. Pages the site leaves out of its sitemaps are not found, and a sitemap can contain stale or broken URLs: this Actor does not visit the pages to check them.
• Sites with no sitemap, or whose sitemap is blocked or behind bot protection, return nothing for that site; the reason is in RUN_SUMMARY and the status message, and nothing is charged for that site.
• It obeys robots.txt for the sitemap files it fetches (rules for *, or rules naming sitemap-url-extractor).
• At most 5,000 sitemap files are read per site per run.
• No personal data is collected: only URLs and the sitemap fields.

Use it from the API, a schedule or an AI agent

API. Start a run and get the results in one call (replace YOUR_APIFY_TOKEN):

curl -X POST "https://api.apify.com/v2/actors/fleet261114~sitemap-url-extractor/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://wordpress.org"], "maxUrlsPerSite": 1000}'

This waits up to 5 minutes. For bigger jobs, start a run with POST /v2/acts/fleet261114~sitemap-url-extractor/runs and read its dataset when it finishes. Apify's JavaScript and Python clients wrap the same calls.

Schedules and integrations. Create a task with your input and attach an Apify schedule (for example daily). Use Apify's integrations (webhooks, Make, Zapier, n8n, Google Sheets and others) to send the results onward.

AI agents (MCP). Add this Actor as a tool in any MCP client (Claude, Cursor and others) through Apify's hosted MCP server:

https://mcp.apify.com?tools=fleet261114/sitemap-url-extractor

The agent sees the input schema and the field descriptions above and can call the Actor directly. Set a maximum cost per run in the call options to cap spending.

FAQ

Does it crawl the website? No. It reads only robots.txt and the sitemap files, never the pages, so it is fast and puts almost no load on the site.

Why did a site return nothing? It has no sitemap at robots.txt or the usual paths, or blocks automated requests. RUN_SUMMARY says which; nothing is charged for that site.

Can I get only recently changed pages? Yes: set "Only URLs modified after". This relies on the site's lastmod values being accurate.

What does a typical run cost? At $0.0005 per URL: 1,000 URLs cost about $0.50; 20,000 URLs about $10. Use the filters, "Max URLs per site" or a maximum cost per run to stay in budget.

Questions and problems

Open an issue on the Actor's Issues tab with the run link.