Sitemap URL Extractor

Returns every URL a website lists in its sitemaps, as a table you can export to CSV, Excel or JSON or read through the API.

Give it a homepage and it looks for the sitemaps itself. It reads the Sitemap: lines in robots.txt first, then tries the usual locations such as /sitemap.xml, /sitemap_index.xml and /wp-sitemap.xml. It follows nested sitemap indexes, unpacks .xml.gz files, and also reads RSS, Atom and plain-text sitemaps.

How it behaves

A broken child sitemap doesn't fail the run. A 404, a timeout or a malformed XML file goes into the per-site SUMMARY record and the Actor moves on. When a file is damaged partway through, you still get the entries before and after the damage.

It uses plain HTTP requests and a streaming parser, with no browser, so runs are fast and the default memory is 512 MB. Large sitemap files are never loaded into memory whole.

URLs that appear in several sitemaps are saved once. URLs removed by your filters aren't saved, and you aren't charged for them.

Each row has lastmod, changefreq and priority when the sitemap provides them. You can also switch on hreflang alternates and image, video and Google News sitemap data.

Status checks are optional. When they're on, the Actor sends one light request per URL and records the HTTP status and any redirect target. It skips URLs that robots.txt disallows and keeps the number of parallel requests per site low.

What people use it for

• SEO audits: pages missing from the sitemap, stale lastmod dates, redirects and broken URLs, hreflang coverage
• Seeding a crawl with a site's full URL list before running another Actor
• Content inventories and site migrations, with every page in one spreadsheet
• Change monitoring: schedule it with "Modified since" set to 7 days and you only get new or updated pages
• Sizing up a competitor's site and the sections it publishes

Input

Field	What it does
Websites or sitemap URLs	Homepages (https://example.com), subfolders (https://example.com/blog/), or direct sitemap or feed URLs. Bare domains work too.
Max URLs per site	Stops after this many saved URLs for one site. 0 means no limit.
Only URLs under the start path	For https://example.com/blog/, keeps only URLs under /blog/. On by default.
Include / Exclude patterns	Case-insensitive regular expressions, for example include /products/ or exclude \.pdf$.
Modified since	Keeps only URLs with a lastmod on or after a date (2024-01-31) or within a period (30 days).
Extra fields	hreflang alternates and image, video and news sitemap data.
Check HTTP status	Adds statusCode, finalUrl and statusError to each row.

Example input:

{
    "startUrls": [{ "url": "https://crawlee.dev" }, { "url": "https://example.com/blog/" }],
    "maxUrlsPerSite": 10000,
    "excludeUrlPatterns": ["\\?", "/tag/"],
    "lastmodSince": "30 days"
}

Output

One row per URL:

{
    "url": "https://crawlee.dev/blog",
    "site": "https://crawlee.dev",
    "sitemapUrl": "https://crawlee.dev/sitemap.xml",
    "lastmod": "2024-04-01T03:00:00.000Z",
    "changefreq": "weekly",
    "priority": 0.5
}

With status checks on, each row also gets statusCode (such as 200 or 404), finalUrl when the URL redirects, and statusError (such as "timed out" or "blocked by robots.txt").

The SUMMARY record in the run's key-value store has one entry per site. It shows how the sitemaps were found (robots.txt, common-path, input or none), how many sitemap files were read, the counts of URLs found, saved, deduplicated and filtered out, and any sitemap errors.

Pricing

You pay per result, with platform compute included. There's no monthly rental.

Event	Price
URL extracted	$0.40 per 1,000 URLs
URL extracted with HTTP status check	$1.00 per 1,000 URLs
Actor start	$0.00005 per run

If you set a maximum cost per run in the run options, the Actor stops cleanly when it reaches it.

FAQ

Is it OK to extract sitemaps? Sites publish sitemaps so that automated tools can find their pages, and this Actor reads only what's published there. Status checks skip URLs that robots.txt disallows. How you use the data is up to you.

What if a site has no sitemap? The SUMMARY record marks it none and there are no URL charges for it, only the per-run start fee. If you know where the sitemap is, give the Actor that URL directly.

Why did I get fewer URLs than I expected? Look at the SUMMARY record. It shows how many URLs were filtered out or skipped as duplicates, any sitemap errors, and whether "Max URLs per site" or "Only URLs under the start path" cut the run short.

Does it render JavaScript or follow links? No, it reads sitemaps only, which is why it's quick and cheap. If you need to crawl the pages, pass its output to a crawler Actor.