Sitemap URL Extractor — extract all URLs from a website

What it does

This fast HTTP-only sitemap extractor turns websites, domains or direct sitemap links into a deduplicated URL inventory. Use it as a sitemap.xml parser, XML sitemap scraper or lightweight sitemap checker, without a browser. It reads the URLs the site publishes, not its HTML navigation.

• Discover Sitemap directives in robots.txt, then try common sitemap paths until one works.
• Follow sitemap indexes with depth and loop protection; read XML, gzip (detected by magic bytes), and plain-text URL lists.
• Extract lastmod, changefreq, priority, image/video counts, news metadata and hreflang alternates.
• Filter URLs with Python regular expressions, date and domain filters; export JSON or CSV from Apify.
• Optionally check URL status with HEAD, falling back to a immediately closed GET on 405/501.

Use cases

• SEO audit: inventory indexed candidates and inspect lastmod and sitemap metadata.
• Content inventory: export a clean URL list to a spreadsheet.
• Site migration: prepare old URL lists for redirect mapping.
• Monitor new pages: run on a daily schedule with lastmodAfter. This filters published dates; it is not a persistent diff database.
• Seed URL lists for crawlers and RAG pipelines, which can fetch content separately.

Input

Field	Default	Meaning
startUrls	https://www.sitemaps.org	Array of website URLs/domains or direct .xml, .xml.gz, .txt links
maxDepth	5	Index depth; root is 0
maxUrlsPerSite	200 in default input	Saved URL cap per start point; set 0 for unlimited
maxUrls	0	Total saved URL cap; 0 is unlimited
maxSitemapsPerSite	500	Maximum attempted files per start point
maxConcurrency	5	Global request limit, maximum 20; per-host concurrency 2
includePatterns	[]	Python regex strings: match any to include; empty includes all
excludePatterns	[]	Python regex strings: any match excludes
lastmodAfter	empty	ISO date/time lower bound, inclusive
excludeMissingLastmod	false	Drop entries without lastmod, independently of date filter
sameDomainOnly	true	Match registrable domain, including subdomains
checkStatus	false	Optional HTTP status and final redirect checks; slower

{
  "startUrls": ["https://www.sitemaps.org", "https://www.wpbeginner.com/sitemap_index.xml"],
  "maxUrlsPerSite": 2000,
  "maxUrls": 3000,
  "maxDepth": 5,
  "lastmodAfter": "2022-01-01",
  "excludePatterns": ["/tag/", "\\?"],
  "sameDomainOnly": true,
  "checkStatus": false
}

Output

One useful dataset item per unique URL across the entire run. Overlapping start points are not billed twice. The first saved listing supplies site and sitemapUrl. Per-site SUMMARY counts describe rows actually saved for that start point.

{
  "url": "https://www.sitemaps.org/protocol.html",
  "lastmod": "2022-12-15",
  "changefreq": null,
  "priority": null,
  "sitemapUrl": "https://www.sitemaps.org/sitemap.xml",
  "site": "https://www.sitemaps.org",
  "imageCount": 0,
  "videoCount": 0,
  "newsTitle": null,
  "newsPublicationDate": null,
  "alternates": [],
  "extractedAt": "2026-11-05T00:00:00+00:00"
}

Field	Meaning
url	Published page URL
lastmod	Date/time normalized to ISO 8601 when parseable, otherwise preserved
changefreq	Published hint or null
priority	Published finite number or null
sitemapUrl	File that first listed the URL
site	Normalized input start point
imageCount / videoCount	Number of image/video entries
newsTitle / newsPublicationDate	News sitemap metadata or null
alternates	Array of {hreflang, href}, empty if absent
statusCode / redirectUrl	Only with checkStatus; null on request failure; redirectUrl null without redirect
extractedAt	UTC extraction timestamp

No billable error rows are saved. Key-value record SUMMARY contains an array of {site, sitemapsFound, sitemapsFailed: [{url, reason}], urls}. Empty or failed sites are logged and no-sitemap sites are mentioned in the final status. Partial malformed files may have already emitted valid preceding URLs, with the parse failure recorded in SUMMARY.

Pricing

You pay only for URLs saved to the dataset, plus a tiny per-run start fee; Apify platform usage is included in the price. Error logs and SUMMARY records are not dataset items. The Actor stops normally when a dataset charge limit is reached. The current price per 1,000 URLs is shown on the Pricing tab of this page.

Tips / limits

Only what the site publishes in sitemaps can be extracted. JS-only sites without sitemaps return nothing. lastmod is the publisher's claim, not proof of freshness; missing and unparseable dates are kept unless missing dates are explicitly excluded. No persistent cross-run deduplication is performed. Invalid start points are skipped with a warning unless all are invalid. SUMMARY status distinguishes completed, stopped and skipped sites. For large inventories, use a practical maxUrls cap: around one million unique URLs per run is a planning ceiling at 256 MB, not a guarantee; long URLs and several near-50 MB sitemaps can need more memory. Raise run memory to 512 MB or split inputs for large sites. Deduplication sets grow with discovered URLs, including filtered-out URLs.

The default is a verified small sitemap at sitemaps.org, capped at 200 saved URLs for cheap daily tests. Set maxUrlsPerSite to 0 to extract an unlimited inventory. Sitemap index depth and file limits still apply. Files are capped at 50 MB uncompressed, including gzip-bomb protection. XML is iteratively parsed; DTD/entity declarations are rejected. Maximum three request attempts, exponential backoff and Retry-After support (waits capped at 30 seconds) apply to transient errors. A site that publishes unusually many duplicate entries still takes time to parse. Filters are Python regex (not globs); avoid expensive pathological expressions.

Compliance & data source

Official public robots.txt and sitemap endpoints only. No login, cookies, proxies, CAPTCHA solving or anti-bot bypass. The sitemap is explicitly published for machines; robots.txt is used for discovery, not to crawl HTML. Optional status checks contact only URLs listed in sitemaps, with HEAD or an immediately closed GET fallback. User-Agent: EchoLab-Apify-Actor/sitemap-url-extractor (+https://apify.com/echolab).

No contact details or private-person fields are collected; sitemap URLs and news metadata are copied as published. You are responsible for ensuring the selected sources and exported metadata comply with applicable terms and privacy rules.

Support

Open an issue on the Actor's Issues tab.

More real output, worked pricing examples and a comparison with alternatives: https://echolab-tools.pages.dev/sitemap-url-extractor/