Sitemap URL Extractor: All URLs, Lastmod, Images & Status

Get every URL listed in a website's XML sitemaps in seconds. Give it a domain and the Actor finds the sitemaps for you (robots.txt Sitemap: lines, then /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, /sitemap.txt), follows nested sitemap indexes, unpacks .xml.gz files and returns one clean row per URL:

• URL, lastmod (normalized to ISO 8601), changefreq, priority
• hreflang alternates (xhtml:link) for multilingual sites
• Image sitemap tags (image URL, title, caption)
• Video sitemap tags (title, description, thumbnail, content/player URL, duration, publication date)
• Google News sitemap tags (headline, publication date, publication name, language)
• Which sitemap file (and parent index) each URL came from
• Optional HTTP status check of every URL (status code and redirect target), to find 404s and redirects in your sitemap

Also reads plain-text sitemaps and RSS/Atom feeds used as sitemaps. No browser, no proxy cost on normal sites, so runs are fast and cheap: 4,800 URLs in about 3 seconds.

Pricing: $0.20 per 1,000 URLs

	This Actor	Typical sitemap extractors on Apify
URL from sitemap	$0.20 per 1,000	$0.50 to $5 per 1,000
URL + status-code check	$0.60 per 1,000	often unavailable or $1+ per 1,000
Actor start	$0.0005 per GB of memory (256 MB default)	up to $0.01

You pay only for URLs you receive. Filters (patterns, lastmod) are applied before charging, so skipped URLs are free. Max URLs per site and Max URLs in total cap every run, and your Apify maximum cost per run is respected exactly.

Use cases

• SEO audits and site migrations: export the full URL inventory of a site, compare old and new sitemaps, find 404s and redirect chains that are still in the sitemap.
• Crawl seeding: feed the URL list into a scraper or crawler (Website Content Crawler, our SEO Audit Tool or Broken Link Checker) instead of discovering pages by crawling.
• Content monitoring: schedule a daily run with lastmodAfter: "1 day" to get only pages a competitor added or changed.
• News monitoring: read Google News sitemaps for fresh headlines with publication times.
• RAG and AI agents: get the list of documentation pages to index.

Input examples

Auto-discover all sitemaps of a website:

{ "startUrls": ["https://crawlee.dev"] }

A specific sitemap, only blog posts changed in the last 30 days:

{ "startUrls": ["https://www.shopify.com/sitemap.xml"], "includePatterns": ["*/blog/*"], "lastmodAfter": "30 days" }

Check every sitemap URL for 404s and redirects:

{ "startUrls": ["example.com"], "checkStatus": true, "maxUrlsPerSite": 5000 }

Output

{
  "type": "url",
  "url": "https://www.bbc.com/news/videos/ckdd22kyq81o",
  "lastmod": "2026-10-29T15:17:00.000Z",
  "changefreq": null,
  "priority": null,
  "alternates": [],
  "images": [],
  "videos": [],
  "news": { "title": "BBC Newsline signed summary", "publicationDate": "2026-10-29T15:17:00.000Z", "publication": "BBC News", "language": "en" },
  "site": "https://www.bbc.com",
  "sitemapUrl": "https://www.bbc.com/sitemaps/https-sitemap-com-news-1.xml",
  "parentSitemapUrl": "https://www.bbc.com/sitemaps/https-index-com-news.xml",
  "statusCode": 200,
  "redirectTo": null,
  "scrapedAt": "2026-10-30T05:17:12.135Z"
}

(statusCode, redirectTo and checkError appear only with Check HTTP status on.)

A per-site SUMMARY record in the key-value store lists every sitemap file found (index or urlset, URL counts) and any sitemap errors. Inputs that fail (no sitemap, DNS error, HTML instead of XML) are listed in FAILURES, and the run keeps going with the other sites.

How it works

• Discovery order: robots.txt Sitemap: directives, then common sitemap paths. A URL that looks like a sitemap (.xml, .gz, .txt, sitemap, feeds) is read directly.
• Sitemap indexes are followed breadth-first, de-duplicated, up to Max sitemap files per site (500 default).
• URLs are de-duplicated per site. http:// sitemaps listed in robots.txt fall back to https:// automatically.
• Polite by default: an identifying user agent, no more than 4 sites in parallel, retries with back-off, and Apify Proxy only if a site refuses direct connections.

FAQ

Does it crawl the website? No. It only reads the sitemaps the site publishes for search engines, which is why it is fast and cheap. Use a crawler if a site has no sitemap.

My site has no sitemap at robots.txt. Put the direct sitemap URL in the input. If nothing is found the input is listed in FAILURES with the reason.

How many URLs can it handle? Hundreds of thousands. Sitemaps are parsed with a lightweight streaming-friendly parser, so 50,000-URL files are fine at 256 MB.

Can I use it from the API, Make, Zapier, n8n or an AI agent (MCP)? Yes, like every Apify Actor. Input is flat JSON, output is a dataset you can download as JSON, CSV or Excel.