Sitemap URL Extractor

Get every URL from a website's XML sitemaps. Enter a website (or a sitemap URL). The Actor finds the sitemaps through robots.txt and the usual locations, follows sitemap indexes and gzipped .xml.gz files, and returns one clean row per page. Each row has lastmod, changefreq, priority, hreflang alternates, image and video counts, and whether robots.txt allows the URL. HTTP status codes are optional.

No browser and no proxies, so it is fast and cheap: $0.30 per 1,000 URLs, status check included.

What it does

• Finds sitemaps automatically. It reads Sitemap: lines in robots.txt, then tries /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml and /sitemap.txt.
• Follows nested sitemap indexes of any depth, and handles gzipped sitemaps (.xml.gz).
• Reads every sitemap format: standard XML, Google image, video and news extensions, xhtml:link hreflang, RSS and Atom feeds, and plain-text URL lists.
• Finds new or updated pages. Filter by lastmod with a date ("2026-01-31") or a period ("7 days"). Child sitemaps older than the cut-off are skipped without being downloaded.
• Include and exclude filters with regular expressions, e.g. only /blog/ or no /tag/.
• robots.txt check for each URL. It flags sitemap URLs that robots.txt blocks, a common SEO mistake.
• Optional HTTP status check. A lightweight HEAD request records the original status code. Redirects are not followed, so 301/302/404 URLs in your sitemap stand out.
• Handles errors well. Retries with backoff on timeouts, 429 and 5xx responses. Unreachable sites or missing sitemaps don't fail the run. Each problem is explained in the SUMMARY record.

Use cases

• SEO audits: find sitemap URLs that redirect, return 404, or are blocked by robots.txt.
• Crawl seeding: give your scraper or RAG/LLM pipeline a complete, de-duplicated URL list instead of crawling link by link.
• Content monitoring: schedule a daily run with lastmodAfter: "1 day" to see a competitor's new pages, products or articles.
• Site migrations: export all old URLs to build redirect maps, then check them afterwards.
• Inventory and research: count pages by section, language (hreflang) or update frequency.

Input

Field	Description	Default
startUrls	Websites (https://example.com) or direct sitemap URLs (.xml, .xml.gz, index, feed, .txt)	required
maxUrlsPerSite	Max URLs saved per start URL (0 = no limit). This also caps your cost.	10000
includePatterns	Keep only URLs that match one of these regexes	–
excludePatterns	Drop URLs that match any of these regexes	–
lastmodAfter	Keep URLs modified on or after a date or period (2026-01-31, 7 days, 3 months)	–
dropUrlsWithoutLastmod	Skip URLs that have no lastmod	false
sameDomainOnly	Keep only URLs on the start URL's domain (www. is ignored)	true
checkStatus	Send a HEAD request to each URL and record its status code and redirect target	false
statusConcurrency	Parallel status requests	5
maxSitemapFiles	Safety limit on sitemap files read per site	500
discoverSitemaps	Try the usual sitemap locations if robots.txt declares none	true

Example input

{
  "startUrls": ["https://apify.com", "https://www.gov.uk/sitemap.xml"],
  "maxUrlsPerSite": 5000,
  "excludePatterns": ["/tag/"],
  "lastmodAfter": "30 days",
  "checkStatus": true
}

Output

Each saved URL is one dataset row. This row is from a real run on https://www.nasa.gov with checkStatus on. Every row has the same fields, with null when the sitemap doesn't provide a value:

{
  "url": "https://www.nasa.gov/directorates/armd/aetc/about-aetc-ground-facilities/",
  "startUrl": "https://www.nasa.gov",
  "sitemapUrl": "https://www.nasa.gov/wp-sitemap-posts-post-1.xml",
  "lastmod": "2026-09-18T12:40:35.000Z",
  "changefreq": null,
  "priority": null,
  "imageCount": 0,
  "videoCount": 0,
  "alternates": [],
  "newsTitle": null,
  "newsPublicationDate": null,
  "robotsAllowed": true,
  "statusCode": 301,
  "redirectLocation": "https://www.nasa.gov/directorates/armd/aetc",
  "statusError": null
}

statusCode, redirectLocation and statusError are filled only when checkStatus is on. A plain date in the sitemap (2026-10-20) is kept as a plain date.

The key-value store record SUMMARY lists, for each start URL: whether robots.txt was found, the sitemaps discovered, how many sitemap files were parsed, URLs found and saved, and a plain-English reason for any problem (e.g. "HTTP 403 (the website refuses automated access to this file)").

You can download the dataset as JSON, CSV, Excel or XML, or get it through the Apify API, webhooks, Zapier, Make or MCP.

Pricing

Pay per event: $0.30 per 1,000 URLs saved ($0.0003 per URL). The HTTP status check costs nothing extra. Sites without a sitemap and URLs removed by your filters cost nothing. Set maxUrlsPerSite or a maximum cost per run to stay within budget; the Actor stops cleanly when the limit is reached.

Examples: a 50-page brochure site costs $0.015. A 2,000-page blog costs $0.60. A 100,000-URL e-commerce sitemap costs $30, or less with filters.

FAQ

Does it crawl the website? No. It reads only the sitemaps the site publishes for robots, plus HEAD requests if you turn on status checking. That makes it fast and light on the target site.

What if a site has no sitemap? The run still succeeds. SUMMARY says that no sitemap was found and you are not charged. For sites without sitemaps, use a link crawler instead.

Why does a site return HTTP 403? Some sites block automated clients. This Actor identifies itself honestly and does not bypass bot protection or use residential proxies, so such sites are reported, not scraped.

My sitemap is on another host (CDN) than my pages. Why are rows missing? sameDomainOnly compares URLs with the start URL's domain. Start from the website address instead, or turn sameDomainOnly off.

Does it respect robots.txt? Yes. Guessed sitemap locations and status checks are skipped for paths that robots.txt disallows. Every row includes robotsAllowed.

How many URLs can it handle? Each sitemap file can be up to 100 MB uncompressed (the protocol limit is 50 MB). URLs are de-duplicated in memory, so for runs with several hundred thousand URLs or more, raise the memory to 1 GB.

Can I get only new pages since my last run? Yes. Schedule the Actor with lastmodAfter set to the period between runs, e.g. "1 day". Accuracy depends on how carefully the site maintains lastmod.

Limits

• Only URLs listed in sitemaps are returned. Pages missing from the sitemap are not discovered.
• lastmod, changefreq and priority are taken as the site declares them.
• Status checks use HEAD (falling back to GET when HEAD isn't supported). A few servers answer HEAD differently from GET.
• No login, cookies or bot-protection bypass.

Questions or a feature request? Open an issue on the Actor's Issues tab.