Sitemap Extractor & URL Discovery

What does this sitemap extractor do?

Sitemap Extractor & URL Discovery finds the URLs a website publishes. Give it domains, homepages or direct sitemap URLs. It reads robots.txt, finds the sitemaps (including nested sitemap indexes, gzipped sitemaps, plain-text sitemaps and RSS or Atom feeds used as sitemaps), and returns one clean record per URL with its lastmod date, priority, hreflang alternates and the sitemap it came from. If a site has no usable sitemap, it can fall back to a shallow, HTTP-only crawl of same-site links from the homepage. Each record says whether the URL came from a sitemap or from that crawl.

It uses plain HTTP requests, no browser, no login and no proxies. It respects robots.txt and waits about a second between requests to the same site by default.

Who is this URL extractor for?

• SEO teams checking which pages a site exposes to search engines and AI crawlers, or what changed recently (filter by lastmod).
• Developers building RAG or crawl pipelines who need a URL list before fetching content.
• AI agents using Apify's MCP server that need to map a site before reading it.

Input

Field	Type	Default	What it does
urls (required)	list	prefill crawlee.dev	Domains (example.com), homepages, direct sitemap URLs or RSS/Atom feed URLs.
maxUrls	integer	1000 (prefill 100)	Total cap on URLs saved. You pay per URL saved, so this also caps cost.
maxUrlsPerSite	integer	0 (no cap)	Stops one big site from using the whole budget.
includeRegex	text	none	Keep only URLs that match, e.g. /blog/.
excludeRegex	text	none	Drop URLs that match, e.g. `/tag/
lastmodAfter	date	none	Keep only URLs modified on or after this date. URLs with no lastmod are dropped when this is set.
fallbackCrawl	boolean	true	Follow same-site links from the homepage if there's no usable sitemap.
maxCrawlPages	integer	30	Pages fetched per site during the fallback crawl.
respectRobotsTxt	boolean	true	Don't fetch sitemaps or pages that robots.txt disallows. Can only be turned off together with ownSiteConfirmed.
checkRobotsForUrls	boolean	true	Adds allowedByRobots to each URL, checked against the robots.txt of the URL's own host. URLs on other hosts cost one extra robots.txt request per host (at most 10 hosts; beyond that null with a robotsNote).
maxSitemapsPerDomain	integer	300	Safety cap on sitemap files fetched per site.
requestDelayMs	integer	1000	Delay between requests to the same site (about 1 request per second, longer if robots.txt sets a Crawl-delay). Shorter delays need ownSiteConfirmed.
maxConcurrency	integer	5	Sites processed in parallel (each site still gets the per-site delay).
ownSiteConfirmed	boolean	false	Tick only for sites you own or are authorised to crawl. Unlocks turning off robots.txt and delays under 1 second.

Example input:

{
  "urls": ["crawlee.dev", "https://docs.apify.com/sitemap.xml"],
  "maxUrls": 100,
  "excludeRegex": "/tag/"
}

Output

One item per unique URL. Here's a real item from a test run (1 Nov 2026):

{
  "url": "https://crawlee.dev/blog",
  "domain": "crawlee.dev",
  "inputUrl": "crawlee.dev",
  "discoveredVia": "sitemap",
  "sourceSitemap": "https://crawlee.dev/sitemap.xml",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "alternates": [],
  "imageCount": 0,
  "allowedByRobots": true,
  "robotsNote": null,
  "discoveredAt": "2026-11-01T20:54:56.778Z"
}

Each run also saves an OUTPUT_SUMMARY record and a SUMMARY.md in the key-value store. Per site, these record whether robots.txt was found, which sitemaps were fetched, any sitemap errors, URLs found versus saved, and whether the sitemap or the crawl was used. Agents and scripts can read them without paging through the dataset.

How much does it cost to extract sitemap URLs?

It's pay per event: $0.40 per 1,000 URLs saved ($0.0004 per URL), charged as one url event for each unique URL saved to the dataset. Example: a 10,000-URL site costs $4.00 in URL events. Any other charges Apify applies are shown on the Pricing tab. Your maxUrls setting and your maximum charge per run both cap the cost, and the run stops when either is reached. Sites or sitemaps that fail or can't be reached save no URLs, so they add no url events.

Use with AI agents

The Actor is designed to run with limited permissions, so AI agents can call it as a tool through the Apify MCP server. Example prompt: "List the blog URLs on crawlee.dev that changed since 1 October."

Limits

• Sitemaps only list what the site owner chose to include. The fallback crawl is shallow (maxCrawlPages), so on sites without a sitemap you may not get every page.
• lastmod is whatever the site reports. Many sites leave it out or set it on every page.
• A sitemap that's blocked, requires a login or returns an error is listed in the run summary as an error. The Actor never works around it.
• Inputs that point at private, loopback, link-local or other reserved network addresses are never requested; the run summary lists them with a blocked-private error. Sitemap entries and redirects to such addresses are refused the same way.
• URLs beyond your caps are not saved. The run summary shows how many were found compared with how many were saved.

FAQ

Does it download page content? No. It returns URLs and sitemap metadata. Pair it with a crawler to get page content. What happens if a site has no sitemap? With fallbackCrawl on (the default), it collects same-site links from the homepage and marks those URLs "discoveredVia": "crawl". With it off, that site returns no URLs, and the run summary's noSitemapReason field says why, for example that robots.txt lists no sitemap and /sitemap.xml returned HTTP 404. The same reason is added to sitemapErrors and the log. Can I monitor new pages? Yes. Schedule it with lastmodAfter set to a date, or compare the URL lists from two runs.

Legal

It collects public URLs and sitemap metadata only, and no personal data. You're responsible for using the results in line with the target site's terms.

Changelog

See the Changelog tab (CHANGELOG.md in the source).