Sitemap URL Extractor

Sitemap URL Extractor pulls every page URL out of one or many websites. Give it a homepage or a sitemap file. It reads robots.txt, checks the sitemap paths sites actually use, follows sitemap indexes, and decompresses .gz files. Each result has the URL, lastmod, changefreq, priority, and the source sitemap.

It uses plain HTTP. It does not start a browser, so a full site map stays fast and cheap enough to run ahead of an SEO audit or a crawl.

Why teams use a sitemap URL extractor

A sitemap is the site's own list of canonical URLs. Crawling every link misses pages that are not linked, and it spends money on pages you did not need. Starting from the sitemap is the usual first step for:

• SEO crawls and content audits. Feed the URLs into an SEO audit and check titles, canonicals, and indexation.
• AI knowledge bases and RAG. Collect the public URL list before you extract page text.
• Change monitoring. Keep only URLs whose lastmod is newer than a date.
• Migration and QA. Diff the URL list before and after a launch.
• Hreflang and media reviews. Read alternate language links, image URLs, video URLs, and Google News fields from the sitemap extensions.

How sitemap discovery works

For a website such as https://example.com, the Actor:

1. Requests /robots.txt and collects every Sitemap: directive.
2. Requests common files the sitemap protocol and popular CMSs use, including /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, news, image, and video sitemap paths, plus .xml.gz variants.
3. If those steps find nothing, it looks for <link rel="sitemap"> on the homepage.
4. Follows sitemap indexes, including nested indexes, and remembers files it has already downloaded.
5. Reads XML sitemaps and plain-text URL lists. Gzip responses and .gz files are decompressed, including a file that was gzipped twice.

A direct sitemap URL such as https://example.com/sitemap.xml is downloaded as-is and is not mixed with other files unless it is an index.

Timeouts, connection errors, HTTP 429, and HTTP 5xx are retried with backoff. A file that still fails is listed in the run summary. The other sitemaps are kept. The run fails only when every sitemap that was found fails, so a silent empty dataset is not treated as success.

Input

{
  "startUrls": [
    { "url": "https://www.python.org" },
    { "url": "https://apify.com/sitemap.xml" }
  ],
  "maxUrls": 1000,
  "includeUrlPatterns": ["/downloads/"],
  "excludeUrlPatterns": ["\\\\?"],
  "minLastmod": "2024-01-01",
  "deduplicate": true
}

Field	Default	What it does
startUrls	required	Website homepages or sitemap URLs. example.com is accepted and treated as HTTPS.
maxUrls	20000	Stop after this many page URLs are saved.
includeUrlPatterns	empty	Case-insensitive JavaScript regular expressions. Empty keeps every URL.
excludeUrlPatterns	empty	Drop URLs that match any of these regular expressions.
minLastmod / maxLastmod	empty	ISO dates. Index entries outside the window are not downloaded.
includeUndated	false	When a date filter is set, also keep URLs that have no lastmod.
deduplicate	true	Keep the first copy of a URL.
includeImages / includeVideos / includeNews / includeAlternates	true	Sitemap extension fields.
useRobotsTxt	true	Read Sitemap: directives.
probeCommonPaths	true	Check well-known sitemap paths.
maxIndexDepth	5	Nested sitemap indexes to follow.
maxConcurrency	8	Parallel downloads.
maxRetries	3	Retries for timeouts and temporary HTTP errors.
requestTimeoutSecs	30	Per-request timeout.
maxSitemapBytes	20000000	Refuse a single response larger than this.

The form opens with maxUrls set to 1000 so a console test stays small. Raise it when you want the whole site.

Output

One dataset item per page URL:

{
  "url": "https://www.python.org/downloads/",
  "lastmod": "2024-10-01",
  "changefreq": "weekly",
  "priority": 0.8,
  "sourceSitemap": "https://www.python.org/sitemap.xml",
  "inputUrl": "https://www.python.org",
  "imageCount": 1,
  "images": [{ "url": "https://www.python.org/static/img/python-logo.png", "title": "Python logo" }],
  "videoCount": 0,
  "newsTitle": null,
  "alternateCount": 2,
  "alternates": [
    { "hreflang": "en", "href": "https://www.python.org/downloads/" },
    { "hreflang": "es", "href": "https://www.python.org/es/downloads/" }
  ]
}

The key-value record SUMMARY lists every sitemap that was fetched, how it was discovered (robots.txt, common-path, direct, sitemap-index, or homepage-link), and the error for any file that failed.

API

curl "https://api.apify.com/v2/acts/appreciative_duct~sitemap-url-extractor/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://apify.com"}],"maxUrls":100}'

Synchronous calls are capped by Apify at about 5 minutes. For a large site, start a run and read the dataset when it finishes.

Pricing

The Actor is free. You pay only the Apify platform usage for compute, storage, and data transfer. It defaults to 256 MB of memory and does not use a browser or a proxy.

The code charges one url-extracted event per saved URL when pay-per-event pricing is turned on. It is not turned on yet.

Tips for large sites

• Set maxUrls while you confirm the right sitemap, then raise it.
• Set minLastmod to skip old sitemap shards. Index files that publish lastmod are not downloaded when they fall outside the window.
• Exclude tag, search, and attachment URLs with a regular expression instead of downloading them into the dataset.
• Turn off image, video, or hreflang fields if you only need the URL list.

FAQ

Where does the sitemap URL extractor look for sitemaps?

It reads Sitemap: lines in robots.txt, then requests the common XML and gzip paths used by custom sites, WordPress, news, image, and video sitemaps. If both are empty, it checks the homepage for link rel="sitemap".

Does it handle sitemap indexes and gzip?

Yes. Indexes are followed up to maxIndexDepth (default 5). Gzip transport compression and .gz files are both decompressed. A gzip file that was compressed twice is decompressed twice.

Can I extract only URLs updated after a date?

Yes. Set minLastmod to an ISO date such as 2024-01-01. URLs without lastmod are skipped unless you enable includeUndated. Child sitemaps whose own lastmod is older are not downloaded.

Why did a website return no URLs?

Open the run summary. Typical causes are: the site has no sitemap, robots.txt and the common paths all returned HTML or 404, or the host blocked the request. The summary names the URL and the HTTP status. Private and link-local addresses are refused so the Actor cannot be pointed at a cloud metadata service.

Does the sitemap extractor run a browser?

No. It is plain HTTP with retries and timeouts. Pages that only appear after JavaScript runs will not be invented. If they are absent from the sitemap, they are absent from the output.

What does a failed sitemap do to the rest of the run?

That file is recorded in SUMMARY with the status code or parse error. URLs from the other files are still saved. The run is marked failed only when every discovered sitemap fails.

How do I audit the URLs this finds?

Run the SEO audit and AI search readiness Actor on the URLs, or pass the same website there and let it read the sitemap itself.

Notes

The Actor sends a normal browser User-Agent and follows a limited number of redirects. It does not execute JavaScript, solve CAPTCHAs, or use a proxy. Respect the site's terms and rate limits. maxConcurrency defaults to 8 so a single site is not flooded.