Sitemap URL Extractor — robots.txt, Sitemap Index, gzip

Get every URL a website publishes in its sitemaps — with lastmod, changefreq, priority, hreflang alternates and image/video/news extensions — as JSON, CSV or Excel, or via API. Paste website URLs (the sitemap is discovered through robots.txt and common paths) or sitemap URLs directly; sitemap indexes and .gz files are followed automatically.

Why this Actor

• Paste a website — the sitemap is found via robots.txt and nine common paths
• Sitemap indexes and gzip files followed automatically
• lastmod, changefreq, priority, hreflang alternates, image/video/news extensions
• Include/exclude regular expressions and a modified-after filter
• Checked every day by an automated run

Sample output

One item of the dataset (long values shortened):

{
  "site": "https://apify.com",
  "url": "https://apify.com/",
  "lastmod": null,
  "changefreq": null,
  "priority": null,
  "sitemapUrl": "https://apify.com/sitemap/pages.xml"
}

What you get

Field	Description
site, sitemapUrl	Where the URL came from
url	The page
lastmod, changefreq, priority	As declared in the sitemap (lastmod normalised to ISO 8601)
alternates	hreflang variants
images, videos, news	Sitemap extensions when present

A per-site summary (how the sitemap was found, sitemaps read, URLs extracted, errors) is saved as SUMMARY.

Input

• Websites or sitemaps — example.com, https://example.com, or https://example.com/sitemap_index.xml.
• Max URLs per site, Max sitemaps per site.
• Include / Exclude URLs matching — regular expressions, e.g. include /blog/, exclude \.pdf$.
• Only URLs modified after — date filter on lastmod.

Limits and notes

• Discovery order: Sitemap: lines in robots.txt, then /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /sitemap/sitemap.xml, /wp-sitemap.xml, /sitemap1.xml, /sitemaps.xml, /sitemap.xml.gz, /sitemap.txt.
• Sitemaps behind bot protection or login return an error for that site; everything else still completes.
• Public data only; the Actor stores nothing beyond the dataset of your run.

Input example

{
  "urls": [
    "https://blog.cloudflare.com"
  ],
  "maxUrlsPerSite": 10000,
  "maxSitemapsPerSite": 200
}

How much does it cost?

Pay per url — no subscription, no minimum, no charge for platform usage.

Volume	Price
1,000 URLs	$0.30
10,000 URLs	$3.00
100,000 URLs	$30.00

The Apify free plan includes $5 of usage every month — about 16,666 URLs with this Actor, no card needed. Nothing else is charged: platform usage is included in the price, and Apify Bronze, Silver and Gold subscribers get 10%, 20% and 30% off these prices.

Use it from code, n8n, Make, Zapier or an AI agent

Run the Actor and download the dataset in one call (JSON by default; add &format=csv or xlsx):

curl -X POST "https://api.apify.com/v2/acts/chorelet~sitemap-url-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://blog.cloudflare.com"], "maxUrlsPerSite": 10000, "maxSitemapsPerSite": 200}'

Python:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("chorelet/sitemap-url-extractor").call(run_input={"urls": ["https://blog.cloudflare.com"], "maxUrlsPerSite": 10000, "maxSitemapsPerSite": 200})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

• n8n, Make, Zapier — use the Apify node/module: run the Actor, then "get dataset items".
• Google Sheets, Slack, webhooks — add an integration on the run's Integrations tab.
• AI agents — the Actor is available as a tool through the Apify MCP server; the dataset schema describes every field for the model.
• Schedules — run it hourly, daily or weekly from the Schedules tab.

FAQ

What if the site has no sitemap?

The Actor reports it for that site and continues with the rest. Try a direct sitemap URL if you know one.

Can I take only part of a site?

Yes — includePattern and excludePattern are regular expressions over the URL, e.g. /blog/ or \.pdf$.

How big can a site be?

Hundreds of thousands of URLs are fine; set maxUrlsPerSite and maxSitemapsPerSite to cap the run.

Does it crawl pages?

No — it reads only the sitemaps, which is why it is fast and cheap. Feed the URLs into a crawler if you need page content.

What does a run cost?

$0.30 per 1,000 URLs. The free plan's $5 a month covers about 16,000 URLs.

Support

Questions, missing fields or a source that changed? Open an issue on the Issues tab or write to support@chorelet.app — problems are usually fixed within a day, and the Actor is checked every morning by an automated test run. If the Actor saved you time, a short review on its Store page helps other people find it.