Sitemap URL Extractor & XML Sitemap Checker

Give it a website or a sitemap URL. It finds the XML sitemaps, follows index files and compressed .gz sitemaps, and lists every page URL with its lastmod, changefreq and priority.

It also does the checks you'd do by hand in an SEO audit: duplicate URLs, missing or badly formatted lastmod dates, URLs pointing at another domain, files over the 50,000 URL limit, sitemaps that won't load, and robots.txt not mentioning the sitemap. Each site gets a pass, warn or fail so you can see the problem ones straight away.

What it does

• Finds the sitemap from robots.txt, falls back to /sitemap.xml, or takes a sitemap link directly
• Follows sitemap index files and opens .gz sitemaps
• Lists every URL with its lastmod, changefreq and priority, plus the oldest and newest dates
• Flags duplicate URLs, bad or missing dates, URLs on other domains and sitemaps that won't load
• Gives each site a pass, warn or fail
• One bad address doesn't stop the rest of the list
• Lets you cap how many URLs come back while still counting them all
• Refuses private and internal addresses

Use cases

• Get a full list of a site's pages to crawl, monitor or migrate.
• Technical SEO audits: broken sitemaps, duplicate URLs, stale or invalid dates.
• See how big a competitor's site is and what they've published lately.
• Feed the URL list into a scraper, link checker or your own pipeline.
• Compare sitemaps before and after a site migration.

How to use

1. Add website addresses or direct sitemap links to URLs.
2. Optionally set maxUrlsPerSite to limit how many URLs come back per site (all of them are still counted).
3. Press Start. You get one row per site with its verdict, issues and URLs.

You can also call it from the Apify API, run it on a schedule, or hook it up to Make, Zapier or n8n. AI agents can use it through Apify's MCP server.

Input

{
  "urls": [
    "https://docs.python.org"
  ],
  "maxUrlsPerSite": 20
}

Output

One dataset item per input with fields: url, source, sitemapsCrawled, isIndex, totalUrls, uniqueUrls, duplicateUrls, urlsWithLastmod, oldestLastmod, newestLastmod, sitemaps, urls, urlsTruncated, verdict, issues, error.

Here's a real result:

{
  "url": "https://docs.python.org",
  "source": "robots.txt",
  "sitemapsCrawled": 1,
  "isIndex": false,
  "totalUrls": 7,
  "uniqueUrls": 7,
  "duplicateUrls": 0,
  "urlsWithLastmod": 0,
  "oldestLastmod": null,
  "newestLastmod": null,
  "sitemaps": [
    {
      "url": "https://docs.python.org/sitemap.xml",
      "type": "urlset",
      "entries": 7,
      "status": 200,
      "error": null
    }
  ],
  "urls": [
    {
      "loc": "https://docs.python.org/3.11/",
      "lastmod": null,
      "changefreq": "yearly",
      "priority": null,
      "sitemap": "https://docs.python.org/sitemap.xml"
    },
    {
      "loc": "https://docs.python.org/3.12/",
      "lastmod": null,
      "changefreq": "yearly",
      "priority": null,
      "sitemap": "https://docs.python.org/sitemap.xml"
    }
  ],
  "urlsTruncated": true,
  "verdict": "warn",
  "issues": [
    "7 URLs without <lastmod>"
  ],
  "error": null
}

Pricing

Pay per event: $0.0016 per site analysed (successful results only; failed items are returned with an error field and are not charged). Set a maximum charge per run in the run options.

So 100 sites cost about $0.16, and 1,000 cost about $1.60.

FAQ

Does it handle sitemap index files and .gz sitemaps?

Yes, it follows indexes and unzips .gz files on its own.

Why aren't image sitemap entries counted as duplicates?

Image, video and news sitemaps are supposed to list pages that are already in the main sitemap, so flagging them would be wrong.

What about really big sites?

Each site gets about 20 seconds. On very large sites you get a summary of the sitemaps it got through in that time, and the result says it stopped early. Raise the run timeout if you need everything.

Do I pay for sites with no sitemap?

No. You get an error row for that site and no charge.

Limitations

• Results reflect what the target returns at run time; they are informational and not a certification of any kind.
• Only public http(s) targets are processed; private and internal addresses are refused.
• Large inputs are capped per run (see input field descriptions).

Other tools you might like

• Website Tech Stack Detector: what a website is built with: CMS, frameworks, analytics, hosting.
• Steam Reviews Analyzer: why players like (or drop) a game, from its Steam reviews.

Support

This Actor is built and maintained by an automated system run by its publisher. If something's wrong, open an issue and it'll be looked at within 7 days. Please don't put passwords or personal details in an issue.