Sitemap URL Extractor: all pages from robots.txt and sitemaps

Give it a domain and get every URL the site publishes in its XML sitemaps, plus lastmod, changefreq, priority, hreflang alternates, image/video counts and Google News titles. No browser is needed: it reads only robots.txt and the sitemap files themselves, so it is fast (about 170,000 URLs in 30 seconds in our tests) and cheap.

What it handles

• Discovery: reads the Sitemap: lines in robots.txt. If there are none, it tries /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap.xml.gz and /sitemap.txt, skipping any path that robots.txt disallows.
• Sitemap indexes, nested up to 6 levels and fetched 4 at a time, with deduplication.
• Gzip (.xml.gz), plain-text sitemaps, RSS/Atom feeds used as sitemaps, and namespace-prefixed XML.
• Extensions: xhtml:link hreflang alternates, image:image, video:video and news:title.
• Filters: include and exclude patterns (*/blog/* globs or /regex/), and modified after a date.
• Only-new mode for monitoring. Schedule it and get only URLs that weren't in earlier runs, e.g. a competitor's new products, articles or landing pages.
• Clear failures. A site with no sitemap, or one that can't be reached, produces one free error row and a SUMMARY entry explaining what was checked.

Input example

{
  "startUrls": ["allbirds.com", "https://www.nytimes.com/sitemaps/new/news.xml.gz"],
  "includePatterns": ["*/products/*"],
  "lastmodAfter": "2026-10-01",
  "maxUrlsTotal": 100000
}

Any page URL works as input; the Actor uses the site's origin. For example, https://wordpress.org/news/ reads all of wordpress.org. To read one sub-sitemap only, pass the sitemap URL directly.

Output

{
  "url": "https://www.nytimes.com/2026/10/27/opinion/ezra-klein-podcast-nate-silver.html",
  "lastmod": "2026-10-27T22:49:48.000Z", "changefreq": null, "priority": null,
  "alternates": null, "imageCount": 1, "videoCount": 0,
  "newsTitle": "Nate Silver on What the Polls Say — and What His Gut’s Telling Him",
  "sitemap": "https://www.nytimes.com/sitemaps/new/news.xml.gz",
  "site": "https://www.nytimes.com", "input": "https://www.nytimes.com/sitemaps/new/news.xml.gz",
  "scrapedAt": "2026-10-28T13:19:32.542Z", "error": null
}

The SUMMARY record in the key-value store shows, per site, how many sitemaps were found and fetched, any sitemap errors, the URLs listed and the URLs output.

Pricing

This Actor uses pay-per-event pricing:

• Apify's standard Actor start fee.
• A per-URL fee for each URL in the dataset.

Error rows are free. Large sites can list millions of URLs (apify.com lists about 820,000), so by default a run stops at 100,000 URLs (maxUrlsTotal). Raise the limit, or set it to 0 for no limit, when you really want everything. You can also set Maximum cost per run, and the Actor stops cleanly at that amount.

Limits and honest notes

• The output is only what the site lists in its sitemaps. Pages missing from the sitemap are not found, because this is not a crawler.
• lastmod values are whatever the site reports, and some sites set every page to "today".
• Only-new mode keys its memory on the exact list of start URLs.

Legal and data use

The Actor fetches only robots.txt and sitemap files. Sitemaps are published so that automated agents can discover a site's URLs. It does not fetch page content, log in or bypass any protection, and it follows robots.txt Disallow rules when it probes for fallback sitemap paths. You are responsible for how you use the URLs and any content you fetch later.