Sitemap & robots.txt Extractor - Every URL a Site Publishes

Get the complete published URL list of any website. The actor reads robots.txt, finds the declared sitemaps, walks sitemap indexes recursively and exports every URL with its last modified date, change frequency and priority.

No API key, no proxy needed.

What you get per URL

Field	Example
url	https://example.com/blog/how-to-migrate
path	/blog/how-to-migrate
lastModified	2026-08-14
changeFrequency	weekly
priority	0.8
imageCount	images declared in the image sitemap extension
alternateLanguages	hreflang codes declared for the URL
sitemapUrl	which sitemap the URL came from

Optionally one extra record with the parsed robots.txt: disallow rules, allow rules and crawl delay.

Input

• website - the site to inspect, e.g. example.com. robots.txt and the conventional sitemap paths are discovered automatically
• sitemapUrls - alternatively, pass sitemap URLs directly if you already know them
• urlContains - keep only URLs containing a string, e.g. /blog/ or /product/
• includeRobotsRules - also emit the parsed robots.txt rules
• maxResults - how many URLs to save, up to 200 000

Example input

{
  "website": "example.com",
  "urlContains": "/blog/",
  "includeRobotsRules": true,
  "maxResults": 5000
}

Use cases

• Technical SEO audits - compare what a site publishes against what is actually indexed
• Content inventory - get every page of a site you inherited, with modification dates
• Site migration planning - build the redirect map from the old URL list before cutover
• Competitor content analysis - see a competitor's entire content footprint and how recently each page changed
• Crawl seeding - feed the URL list into a scraper instead of discovering links page by page
• Freshness monitoring - lastModified shows which sections a competitor actually maintains

Why start from the sitemap

A sitemap is the site's own declaration of what matters: it is complete by design, already deduplicated, and carries modification dates that no crawl can infer. Starting there is dramatically cheaper than crawling, and it finds pages that no internal link points to.

Sitemap indexes are the normal shape for large sites, so recursive walking is the difference between 50 URLs and 50 000.

Notes

• Sitemap indexes are followed automatically, up to 300 sitemap files per run.
• If robots.txt declares no sitemap, the conventional locations sitemap.xml, sitemap_index.xml and sitemap-index.xml are tried.
• lastModified, changeFrequency and priority are optional in the sitemap standard, and many sites omit them or fill them in mechanically. Treat them as hints.
• A run that finds nothing fails with an explicit message rather than reporting an empty success, so a missing or blocked sitemap is never silent.