Sitemap URL Extractor: All URLs, Status Codes & New Pages

Get every URL of any website from its XML sitemaps in seconds, as a clean CSV, Excel or JSON list. Paste a domain and the Actor finds the sitemaps itself (robots.txt, sitemap.xml, sitemap_index.xml, WordPress, Shopify...), follows nested sitemap indexes, unzips .xml.gz, reads RSS/Atom feeds and text sitemaps, and gives you one row per URL with lastmod, changefreq, priority and the sitemap it came from.

Turn on status checks to find 404s, redirects and noindex headers among the URLs your sitemap promotes, or turn on monitoring to get only the new, changed or removed URLs since the last run: new products, articles, job pages or landing pages on any site, on a schedule. Failed sitemaps are free, and the Actor stops before going over your max cost.

What it does

• 🔎 Sitemap auto-discovery: Sitemap: lines in robots.txt, then /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap.txt. For https://example.com/blog/ it first tries sitemaps under /blog/. Or paste any sitemap or feed URL directly.
• 🗂️ Every sitemap format: <urlset>, nested <sitemapindex> (configurable depth), gzip files, RSS 2.0 and Atom feeds, plain-text URL lists. Malformed XML (unescaped &, truncated files) is recovered instead of failing.
• 🖼️ Sitemap extensions: image URLs, video titles/content/thumbnail URLs, Google News titles and publication dates, and hreflang alternates.
• 🧹 Filters: include/exclude patterns (text, * wildcards or /regex/), modified after/before (2026-01-01 or 7 days), same-site only, max URLs per site and in total. Duplicate URLs across sitemaps are removed.
• 🚦 Status checks (optional): HTTP status, final URL, redirect chain, content type, X-Robots-Tag and response time for each URL. HEAD first with automatic GET fallback; retries for timeouts, 5xx and 429. Save only broken URLs for a quick sitemap audit.
• 🔔 Monitoring: only new URLs: remembers each site's URLs between runs and outputs only new ones, or new + updated (newer lastmod), plus optionally removed URLs. Schedule it and connect an integration (email, Slack, webhook, Google Sheets, Zapier, Make) to get alerts.
• 🔗 Integration friendly: read websites from another Actor's dataset ({{resource.defaultDatasetId}}), runs with limited permissions, stable error codes, a run summary in the OUTPUT record.
• 🛡️ Safe and polite: one global requests-per-second limit, honest User-Agent, private/internal addresses blocked, no browser and no login needed.

Pricing

Pay per event. Compute and storage are included.

Event	Free & Bronze	Silver	Gold & above
URL saved to the dataset	$0.00015 ($0.15 per 1,000)	$0.0001	$0.00008
Status check (only when "Check status codes" is on)	$0.0002 ($0.20 per 1,000)	$0.00015	$0.00012
Run start	$0.00005 per GB of memory (512 MB default = 1 event)	same	same

Not charged: sitemaps that fail to download or parse, URLs removed by your filters, duplicate URLs, unchanged URLs in monitoring mode, and status checks that end in a network error (DNS, timeout, refused connection) or a 429 rate limit after retries.

Examples (Free/Bronze prices):

Job	Cost
Full URL list of a 5,000-page site	about $0.75
1,000 URLs from a Shopify store's product sitemaps, modified in the last 90 days	about $0.15
Sitemap audit of 10,000 URLs with status checks, saving only the 40 broken ones	about $2.01
2,000 URLs with status codes and redirects	about $0.70
Daily "new pages" monitor of 3 competitor sites, ~20 new URLs a day	about $0.10 a month

Set Max cost per run in the run options: the Actor checks the budget before saving or checking URLs and stops cleanly when it would go over.

Running with empty input starts a free offline demo (a sample site with a sitemap index, 2 sitemaps, 7 URLs and status checks; nothing is charged), so you can see the output first.

How to use it

List every URL of a website

1. Add the website in Websites or sitemap URLs (https://www.example.com), or paste bare domains in Domains or URLs (plain text).
2. Start the run. Download the list from the Output tab as CSV, Excel or JSON, or use the plain URL list link (one URL per line).

{ "urls": ["crawlee.dev", "allbirds.com"] }

Only some sections, only recent pages

{
  "startUrls": [{ "url": "https://www.example.com" }],
  "includeUrlPatterns": ["/blog/", "/guides/*"],
  "excludeUrlPatterns": ["/tag/", "/author/"],
  "lastmodAfter": "30 days"
}

Find broken URLs in a sitemap

{
  "startUrls": [{ "url": "https://www.example.com/sitemap_index.xml" }],
  "checkStatus": true,
  "onlyBroken": true
}

Each row gets status, isBroken, finalUrl, redirectCount, redirectChain, contentType, xRobotsTag and responseMs. URLs that redirect are not marked broken, but redirectCount > 0 shows sitemap entries you should update.

Monitor sites for new pages

{
  "urls": ["competitor-one.com", "competitor-two.com/blog/"],
  "monitorMode": "new",
  "monitorName": "competitors",
  "silentFirstRun": true
}

Create a schedule (daily or hourly) for this input. The first run just remembers the current URLs (with "Silent first run"), and every later run outputs only URLs that weren't there before, with changeType: "new". Use newAndChanged to also get URLs whose lastmod got newer (changeType: "changed", with previousLastmod), and Also report removed URLs for pages that disappeared. Removals are only reported when the whole sitemap was read without errors or limits, so a failed download never looks like a mass deletion.

The memory lives in a key-value store named sitemap-monitor-<Monitor name> on your account. Use different monitor names for different jobs, or Reset memory to start over.

Websites from another Actor (integration)

In a task or integration, set Input dataset to {{resource.defaultDatasetId}} and URL field to the field that holds the website (url, website, company.website). Each row's site is processed once.

Output

One item per URL:

{
  "url": "https://www.example.com/blog/sitemap-seo-checklist",
  "lastmod": "2026-10-28T09:30:00.000Z",
  "changefreq": "weekly",
  "priority": 0.8,
  "site": "https://www.example.com",
  "sitemap": "https://www.example.com/sitemap-blog.xml",
  "imageCount": 1,
  "images": ["https://www.example.com/img/checklist.png"],
  "alternates": [{ "hreflang": "de", "url": "https://www.example.com/de/blog/sitemap-seo-checklist" }],
  "changeType": "new",
  "status": 200,
  "isBroken": false,
  "finalUrl": "https://www.example.com/blog/sitemap-seo-checklist",
  "redirectCount": 0,
  "contentType": "text/html; charset=utf-8",
  "responseMs": 184,
  "checkedWith": "HEAD"
}

Fields that don't apply are left out (no status fields without status checks, no changeType without monitoring). RSS/Atom items also get title; news sitemaps add news.title, news.publicationDate, news.publication; video sitemaps add videos.

The OUTPUT record has a run summary: sites, sitemap files read and failed, URLs found/saved/filtered/duplicate, monitor counts (new, changed, unchanged, removed), status codes by count, per-site details (how the sitemap was found) and export links.

Error codes

Problems are listed in the ERRORS record and in OUTPUT.failures with a code. A run succeeds when at least one sitemap was read (turn on Fail the run on any error for strict integrations).

Code	Meaning
NO_SITEMAP_FOUND	robots.txt has no Sitemap: line and none of the usual locations exist. Paste the sitemap URL directly.
HTTP_403, HTTP_401	The site refused access, often because it blocks datacenter IPs. Try Proxy configuration.
HTTP_404, HTTP_410	A sitemap listed in robots.txt or an index doesn't exist.
HTTP_429, HTTP_5xx	Rate limit or server error that persisted after retries.
NOT_A_SITEMAP	The file is HTML or another format, not a sitemap or feed.
BAD_XML, BAD_GZIP	The file couldn't be parsed or unzipped, even with recovery.
TOO_LARGE	The file is bigger than Max sitemap file size.
TIMEOUT, CONNECTION, DNS, TLS	Network problems (timeouts and connection resets are retried).
BLOCKED_HOST	The URL points to a private or internal address.
INPUT, DATASET_ACCESS	Invalid input, or the input dataset can't be read.

Status checks use checkError: RATE_LIMITED (429 after retries, not charged, status unknown) or a network code above.

FAQ

Does it crawl the pages? No. It reads only sitemaps (and, with status checks, sends one lightweight HEAD/GET per URL). That's why it's fast and cheap: 50,000 URLs take about 10 seconds.

What if a site has no sitemap? You get NO_SITEMAP_FOUND and nothing is charged. A crawler is the right tool for such sites.

How big can a site be? Hundreds of thousands of URLs work at the default 512 MB. For millions of URLs in monitoring mode, use 1-2 GB of memory.

Is it legal? Sitemaps are public files that sites publish specifically so that machines can read them. The Actor reads only public URLs, needs no login and collects no personal data. Respect each site's terms when you use the results.

Can I get just a plain list of URLs? Yes: the "Plain URL list" link in the Output tab, or .../items?format=csv&fields=url&skipHeaderRow=1.

Questions or a sitemap that doesn't parse? Open an issue on the Actor's Issues tab with the sitemap URL.