Sitemap URL Extractor & Status Checker

Get every URL a website publishes in its sitemaps as a clean, deduplicated list — with lastmod, changefreq, priority, image counts and hreflang alternates. Optionally check every URL for HTTP status, redirects and noindex headers.

Enter a website (example.com) or a sitemap URL. The Actor finds the sitemaps for you, follows nested sitemap indexes, and handles gzipped, plain-text, RSS and Atom sitemaps.

What you can use it for

• SEO audits – find URLs in your sitemap that return 404, redirect or are marked noindex.
• Site migrations – get before/after URL inventories and compare them.
• Content inventories – list every blog post or product page with its last-modified date.
• Crawl seeding – feed a complete URL list into another scraper, link checker or AI agent.
• Monitoring – schedule a run to see which pages were added or updated.

Features

• Auto-discovery from robots.txt Sitemap: lines, then common locations (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap.txt).
• Nested sitemap indexes followed automatically, with a safety cap on the number of files.
• All sitemap formats: XML urlset/sitemapindex, .xml.gz, plain-text, RSS 2.0 and Atom.
• Image and hreflang extensions parsed (imageCount, optional images, alternates).
• Filters: include/exclude by substring or /regex/. Entering example.com/blog/ keeps only URLs under /blog/.
• Optional status check: one HEAD request per URL (GET fallback if HEAD is rejected), with status code, full redirect chain, final URL, X-Robots-Tag and response time, plus issues tags (http-404, redirect, redirect-chain, noindex-header, unreachable).
• Robust: retries with backoff (honours Retry-After), timeouts, gzip-bomb and size limits, and clear per-site error reports. One broken site never stops the others.

Input

Field	Default	Description
startUrls	–	Websites or sitemap URLs (required).
maxUrlsPerSite	10000	Stop after this many saved URLs per site (0 = no limit). Also caps your cost.
includePatterns / excludePatterns	[]	Substrings or /regex/.
checkStatus	false	Request each URL and record status/redirects/noindex.
restrictToInputPath	true	For example.com/blog/, keep only URLs under /blog/.
maxSitemapFiles	500	Safety cap on sitemap files per site.
includeImages	false	Add image URLs to each row.

Example:

{
  "startUrls": ["https://www.sitemaps.org", "https://crawlee.dev/sitemap.xml"],
  "maxUrlsPerSite": 5000,
  "excludePatterns": ["/tag/"],
  "checkStatus": false
}

Output

One row per unique URL:

{
  "site": "https://www.sitemaps.org",
  "url": "https://www.sitemaps.org/protocol.html",
  "lastmod": "2022-12-15",
  "changefreq": null,
  "priority": null,
  "imageCount": 0,
  "sitemapUrl": "https://www.sitemaps.org/sitemap.xml",
  "sitemapType": "urlset"
}

With checkStatus: true each row also has status, ok, finalUrl, redirectCount, redirectChain, responseTimeMs, xRobotsTag, error and issues.

A run summary (sitemaps found, failed sitemap files with reasons, truncation) is saved as OUTPUT in the run's key-value store. You can download results as JSON, CSV, Excel or HTML, or read them through the Apify API.

How much does it cost to extract sitemap URLs?

Pay per event. You only pay for URLs saved to your dataset, plus a tiny Actor start fee ($0.00005 per GB of memory):

• URL extracted: $0.0003 per URL ($0.30 per 1,000) when the status check is off.
• URL checked: $0.001 per URL ($1.00 per 1,000) when the status check is on.

Sites without a readable sitemap, failed sitemap files and filtered-out URLs are not charged. Platform usage is included in the price. To cap spending, use maxUrlsPerSite or the run's maximum cost setting; when the cap is reached, the run stops cleanly.

Limitations and fair use

• It reads only sitemap files, which websites publish for machines to read. It does not crawl pages or render JavaScript. With checkStatus on, it sends one lightweight request per URL.
• Sites behind aggressive bot protection may block sitemap downloads. Such sites are reported in OUTPUT and not charged.
• Some sites list URLs in sitemaps that no longer exist. That is exactly what checkStatus is for.
• Please use status checks on sites you own or manage, or at a gentle concurrency.

FAQ

The site has no sitemap. What happens? The run reports it in OUTPUT and the status message, and costs nothing beyond the Actor start.

Can I pass several sites? Yes, up to 1,000 per run. Each site is deduplicated separately.

Does it support sitemap.xml.gz? Yes. Gzip is detected by content, not file name.

Use it from AI agents (MCP), the API, Make, Zapier and n8n

• AI agents and MCP clients (Claude, ChatGPT, Cursor, VS Code): add https://mcp.apify.com?tools=pinkish_armada/sitemap-url-extractor as an MCP server, and your agent can list every URL of a website from its sitemaps and check for 404s as a tool call.
• API: run it and get the results in one request:

curl -X POST "https://api.apify.com/v2/acts/pinkish_armada~sitemap-url-extractor/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":["https://www.sitemaps.org"],"maxUrlsPerSite":1000}'

• No-code: use the official Apify apps for Make, Zapier and n8n ("Run an Actor"), then send results to Google Sheets, Slack, Airtable or your database. You can also schedule runs in Apify Console and add a webhook or email integration.

Related tools

• Broken Link Checker: crawl a website and report broken links, 404s and redirects
• PDF to Markdown: turn PDFs (including scans, via OCR) into Markdown, text and tables
• Office Docs to Markdown: turn DOCX, PPTX, XLSX, HTML and EPUB into clean Markdown for LLMs and RAG
• Domain Lookup: bulk RDAP WHOIS, DNS, SPF/DMARC and SSL-expiry checks for domains