Sitemap URL Extractor: Get All URLs From Any Website

Who it's for: SEO specialists, agencies, developers and data/AI teams who need a website's full URL list without crawling it.

The problem it solves: Crawling a whole site just to list its pages is slow and expensive. Sitemaps already list them, but they are split across indexes, gzipped files and odd formats. This Actor reads all of them and gives you one clean, de-duplicated URL list.

Use cases:

• SEO audits: find stale, orphan or unexpected URLs with lastmod
• Build input lists for crawlers, PageSpeed audits, link checkers or RAG/LLM ingestion
• Site migrations: compare old vs new URL inventories
• Monitor competitors' new pages on a schedule

Quick example: {"startUrls":[{"url":"https://crawlee.dev"}]} → one row per URL: url, lastmod, changefreq, priority, sourceSitemap.

Price: $1.00 per 1,000 URLs on the Free plan, as low as $0.23 on higher plans. Comparable sitemap Actors in the Store charge $0.50 to $30 per 1,000 results on the Free plan. Discovering sitemaps, duplicates and errors are free.

Get a clean list of every URL a website publishes in its sitemaps, with lastmod, changefreq, priority and the sitemap each URL came from. Give it a homepage and it finds the sitemaps for you. Give it a sitemap and it follows every child sitemap. No browser and no proxies, so it's fast and cheap: tens of thousands of URLs per minute.

What it does

• Finds sitemaps automatically. It reads Sitemap: lines from robots.txt. If there are none, it tries common locations: /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap.txt.
• Follows sitemap indexes to any depth (WordPress, Yoast, Rank Math, Shopify, Wix, Next.js, Docusaurus and so on).
• Reads every common format: XML urlset, sitemapindex, gzipped .xml.gz (detected from the file contents, not the name), plain-text sitemaps, and RSS / Atom feeds.
• Collects extra details: image count, video and news flags, and the number of hreflang alternates for each URL.
• Filters: max URLs, include/exclude regex, and a same-domain-only option.
• Handles errors: timeouts, retries with backoff on 429 / 5xx / network errors, removal of duplicate URLs, and a size guard for very large files. A full run summary, including any errors, is saved to the SUMMARY record in the key-value store.

Use cases

• SEO audits: find orphan, stale or unexpected URLs, and check lastmod freshness
• Prepare URL lists for crawlers, scrapers, Lighthouse/PageSpeed audits or link checkers
• Content inventories and migration checks (old vs new site)
• Monitor competitors' new pages: schedule a run and compare datasets
• Feed URLs into other Apify Actors

Input

Field	Type	Default	Description
startUrls	array	https://crawlee.dev	Website URLs (sitemaps discovered automatically) or direct sitemap URLs
maxUrls	integer	1000	Maximum URLs to output (0 = no limit; the run's max cost still applies)
includeRegex	string	–	Keep only URLs matching this regex (case-insensitive)
excludeRegex	string	–	Drop URLs matching this regex
sameDomainOnly	boolean	true	Drop URLs on other hosts (www. is ignored)
discoverCommonPaths	boolean	true	Try common sitemap paths when robots.txt lists none
maxSitemaps	integer	500	Safety cap on sitemap files fetched
maxConcurrency	integer	10	Sitemap files downloaded in parallel
requestTimeoutSecs	integer	30	Timeout per request (each request is retried up to 3×)

Example input

{
  "startUrls": [
    { "url": "https://wordpress.org" },
    { "url": "https://developer.mozilla.org/sitemap.xml" }
  ],
  "maxUrls": 20000,
  "includeRegex": "/docs/|/blog/",
  "excludeRegex": "\\.(pdf|jpg|png)$",
  "sameDomainOnly": true
}

Output

One dataset item per unique URL:

{
  "url": "https://crawlee.dev/blog",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "sourceSitemap": "https://crawlee.dev/sitemap.xml",
  "sitemapType": "urlset",
  "sitemapDepth": 0,
  "inputUrl": "https://crawlee.dev/",
  "imageCount": 0,
  "hasVideo": false,
  "hasNews": false,
  "alternateLanguageLinks": 0
}

lastmod and similar fields are passed through exactly as the site publishes them. null means the sitemap doesn't include that field. You can download the results as JSON, CSV, Excel, XML or HTML, or get them through the API.

The run summary (SUMMARY in the key-value store) looks like this:

{
  "inputs": 4, "sitemapsDiscovered": 297, "sitemapsFetched": 63, "sitemapsFailed": 0,
  "urlsFound": 73640, "urlsFiltered": 52434, "duplicates": 1206, "urlsSaved": 20000,
  "urlsPerInput": { "https://wordpress.org/": 12581, "https://developer.mozilla.org/": 7419, "https://www.python.org/": 0 },
  "inputsWithoutUrls": ["https://www.python.org/"],
  "durationSecs": 14.7, "errors": []
}

Pricing

Pay per result: from $1.00 per 1,000 URLs ($0.001 per URL) on the Free plan, with lower prices on paid Apify plans:

Apify plan	Price per 1,000 URLs
Free	$1.00
Starter (Bronze)	$0.63
Scale (Silver)	$0.45
Business (Gold) and above	$0.23

Plus the standard Apify "Actor start" event ($0.00005 per run). You only pay for URLs saved to your dataset. Discovering and downloading sitemaps, filtered-out URLs, duplicates and errors are free. Platform usage (compute, storage) is included in the price. Set a maximum cost per run and the Actor stops cleanly when it reaches that limit. Example: 10,000 URLs on the Free plan cost about $10.00.

Performance

In local tests, the default input took about 1 second of work (1,000 URLs from 2 sitemaps). A mixed run of 4 sites crawled 63 sitemaps, read 73,640 URLs and saved 20,000 in about 15 seconds at 256 MB memory.

Related Actors

More no-login, pay-per-result tools from the same developer:

• Bulk PageSpeed Insights & Lighthouse Audit: Lighthouse scores and Core Web Vitals for a list of URLs.
• Whole-Site PageSpeed Audit (Sitemap → Lighthouse): enter a homepage and get PageSpeed audits of its pages plus a site summary.
• SEO Meta Tags & Structured Data Auditor: titles, descriptions, canonical, noindex, Open Graph and Schema.org rich-result checks per URL.
• Wayback Machine Snapshot Finder: archived snapshots, first/last capture and every archived URL of a domain.
• Domain Health Checker: WHOIS, DNS & SSL: domain age and expiry, DNS/SPF/DMARC, SSL certificate and HTTPS redirects.
• ATS Jobs Scraper: open jobs from Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Recruitee and Personio.

Use together

1. Run Sitemap URL Extractor with "includeRegex": "/blog/" and "maxUrls": 50.
2. Pass the dataset's url column to Bulk PageSpeed Insights & Lighthouse Audit as urls, either by hand, with an Apify integration ("Run another Actor when this one succeeds"), or in code:

const { defaultDatasetId } = await client.actor('aaronallday89/sitemap-url-extractor').call({ startUrls: [{ url: 'https://example.com' }], includeRegex: '/blog/', maxUrls: 50 });
const { items } = await client.dataset(defaultDatasetId).listItems({ fields: ['url'] });
await client.actor('aaronallday89/pagespeed-bulk-audit').call({ urls: items.map((i) => ({ url: i.url })), strategy: 'mobile' });

FAQ

The site has no sitemap. What happens? The run still succeeds with 0 URLs for that input. The input is listed in SUMMARY.inputsWithoutUrls and a warning appears in the log. This Actor only reads sitemaps and doesn't crawl links. If you need link crawling, use a crawler such as Website Content Crawler.

Does it support gzipped sitemaps? Yes. Any gzip file is detected from its first bytes and decompressed automatically, whatever its name or headers say.

Can I get URLs from subdomains or CDNs listed in the sitemap? Yes. Turn off sameDomainOnly.

Why are there fewer URLs than in the sitemap? Duplicates (the same URL in several sitemaps) are removed, and your filters and maxUrls apply. SUMMARY shows urlsFound, urlsFiltered and duplicates.

Does it use proxies or a browser? No. It makes plain HTTP requests with a polite user agent, which keeps it fast and inexpensive. Sitemaps are public files that sites publish so they can be read.

How fresh is lastmod? It's exactly what the website publishes. Some CMSs update it on every build, so treat it as a hint.

Is it legal? Sitemaps are published so machines can read them. You're still responsible for how you use the data. Respect each website's terms of service and applicable law.

Feedback

Found a sitemap it can't read? Open an issue on the Actor's Issues tab with the URL and it will be fixed.

More tools from the same developer

• Bulk PageSpeed Insights Checker: Lighthouse & Core Web Vitals: SEO agencies, web developers, site owners and performance teams
• Whole-Site PageSpeed & Core Web Vitals Audit (via Sitemap): SEO consultants, agencies and site owners
• Greenhouse, Lever, Ashby & Workable Jobs Scraper (7 ATS): Job boards and aggregators, recruiters, sales teams prospecting on hiring signals, and market researchers
• Bulk WHOIS, DNS & SSL Expiry Checker: Domain Health: IT admins, agencies managing client domains, email deliverability teams, domain investors and lead-enrichment workflows