Sitemap URL Extractor: get every URL from any website's sitemap

Extract every page URL from a website's XML sitemaps in seconds, with lastmod, changefreq, priority, image counts and hreflang counts. Sitemaps are found automatically through robots.txt, /sitemap.xml and /sitemap_index.xml. Nested sitemap indexes, .xml.gz and plain-text sitemaps all work.

Use it for SEO audits, content inventories, competitor new-page monitoring, migration checks, and feeding crawlers or RAG pipelines.

Features

• Sitemap discovery: robots.txt Sitemap: lines, plus the common default paths.
• Recursive: follows sitemap indexes, with a maxSitemaps safety cap.
• Filters: include and exclude regexes, and lastmodAfter for recently changed pages.
• New-page monitor: onlyNewSinceLastRun outputs only URLs you haven't seen before. Schedule it daily.
• No paying for junk: duplicate URLs across sitemaps are removed before output. If no sitemap can be read, the run fails loudly and nothing is charged.
• Hard caps: maxUrls, per-request timeout, 3 retries.

Use cases

• SEO audits: get the full URL inventory of a site (or a competitor's) with lastmod dates.
• Competitor content monitoring: turn on onlyNewSinceLastRun and schedule daily to see every new page they publish.
• Crawl seeding: feed exact URL lists into a content crawler or screenshot tool instead of blind link-following.
• Migrations: build redirect maps by comparing old and new sitemaps.

Try it free in 1 click

The input form is prefilled with a small, working example. Press Start without changing anything: the run extracts 50 URLs from the crawlee.dev sitemap. Our test run of exactly this input finished in about 2 seconds with 50 results. At list price that is about $0.01, which fits inside the free monthly credit of Apify's free plan. Then swap in your own values.

Prefilled input:

{
  "sites": [
    "https://crawlee.dev"
  ],
  "maxUrls": 50
}

Sample output

Real rows from the prefilled run on 2026-10-14 (5 of 50 results shown; long fields trimmed). Download the full dataset as JSON, CSV or Excel.

url	path	changefreq	priority	lastmod
https://crawlee.dev/blog	/blog	weekly	0.50	–
https://crawlee.dev/blog/archive	/blog/archive	weekly	0.50	–
https://crawlee.dev/blog/authors	/blog/authors	weekly	0.50	–
https://crawlee.dev/blog/building-gradcracke…	/blog/building-gradcracker-scraper-with-crawlee	weekly	0.50	–
https://crawlee.dev/blog/common-problems-in-…	/blog/common-problems-in-web-scraping	weekly	0.50	–

Input (copy-paste)

{
  "sites": ["https://crawlee.dev", "https://www.allbirds.com/sitemap.xml"],
  "includePatterns": ["/docs/", "/products/"],
  "lastmodAfter": "2026-01-01",
  "maxUrls": 10000
}

API (copy-paste)

curl -X POST "https://api.apify.com/v2/acts/appetizing_horsefly~sitemap-url-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' -d '{"sites":["https://crawlee.dev"],"maxUrls":500}'

Output

{"url":"https://crawlee.dev/js/docs/quick-start","lastmod":"2026-09-30","changefreq":"weekly","priority":0.5,"imageCount":0,"sitemapUrl":"https://crawlee.dev/sitemap.xml","site":"https://crawlee.dev"}

A per-site report (sitemap files read, errors, caps hit) is saved as SUMMARY.

Full JSON of one result (real run)

{
  "url": "https://crawlee.dev/blog",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "imageCount": 0,
  "alternateLanguages": 0,
  "sitemapUrl": "https://crawlee.dev/sitemap.xml",
  "site": "https://crawlee.dev",
  "path": "/blog",
  "extractedAt": "2026-10-12T17:47:29.409Z"
}

Pricing

Pay per event: $0.0003 per URL ($0.30 per 1,000 URLs). Duplicates, filtered and already-seen URLs are free.

Use it as an MCP tool (Claude, Cursor, ChatGPT, any AI agent)

This Actor works as a tool in the Apify MCP server, so an AI agent can call it directly and you pay per result, the same as a normal run. Add this server URL to any MCP client:

https://mcp.apify.com/?tools=appetizing_horsefly/sitemap-url-extractor

Sign in with your Apify account (or pass your Apify API token as a Bearer header). The agent reads the input fields above from the Actor's input schema. Example request to the agent: "List every URL in crawlee.dev's sitemap that was modified in the last 30 days."

Support

Questions, a bug, or a data source you need? Use this Actor's Issues tab or email gm-labs@mail.grokbot.com. Built and maintained by GM Labs.