Sitemap Extractor Pro: Extract All URLs from Any Website's Sitemap

Sitemap Extractor Pro finds and reads every sitemap of any website and gives you a clean list of all its URLs. Each URL comes with lastmod, changefreq, priority, image, video and Google News data and hreflang alternates.

Give it a domain. It discovers the sitemaps on its own: robots.txt, WordPress, Yoast, Shopify, Wix, Squarespace, gzip files, nested sitemap indexes, RSS/Atom feeds and plain-text sitemaps. You can filter by date or regex, and optionally check the HTTP status of every URL to find broken pages and redirects.

It is built to never fail your run because one website is broken. A site with no sitemap, broken XML, a timeout or a 403 is reported clearly in the run summary, and every other site in your batch still gets extracted.

What can it do?

• Automatic sitemap discovery. It checks:
  • robots.txt Sitemap: lines
  • about 20 common and CMS paths (/sitemap.xml, /sitemap_index.xml (Yoast, Rank Math, AIOSEO), /wp-sitemap.xml (WordPress core), /sitemap.xml.gz, /sitemap.txt, news sitemaps, Magento/Drupal/PrestaShop paths)
  • homepage <link rel="sitemap"> tags
  • www/non-www and https/http variants
• Every sitemap format. XML urlset, sitemap indexes nested to any depth (loops are detected), .xml.gz gzip (even when the server double-compresses it or labels it wrongly), RSS 2.0, RSS 1.0/RDF, Atom and plain-text URL lists.
• Tolerant parsing. Broken XML is common: unescaped &, unclosed tags, truncated or corrupt gzip, wrong encodings, an HTML "soft 404" served as sitemap.xml. The Actor recovers every URL it can and tells you what was wrong.
• Rich output.
  • url, lastmod (normalised to ISO 8601), changefreq, priority
  • which sitemap the URL came from and how that sitemap was found
  • image:image entries, video:video entries, news:news (publication, language, date, headline) and xhtml:link hreflang alternates
• Feed fallback. If a site has no XML sitemap, URLs are taken from its RSS/Atom feed.
• Filters.
  • "Changed since" with an absolute date or a relative period like 7 days. Child sitemaps that haven't changed are skipped too, which makes it much faster on big sites.
  • include/exclude regex
  • same-domain only
  • de-duplication
  • max URLs, overall and per site
• Optional HTTP status check.
  • HEAD request with an automatic GET fallback, so servers that reject HEAD don't produce false "broken" results
  • records the first status code, the redirect count, the final URL and the final status
  • polite concurrency limits per site
• Reliable and polite.
  • retries with exponential backoff and Retry-After
  • timeouts for stalled servers
  • an honest User-Agent
  • respects robots.txt when fetching sitemaps
  • per-host concurrency limits
  • optional proxy

Who is it for? Use cases

• SEO audits. List every indexable URL, find sitemap URLs that return 404, 5xx or redirects, and check lastmod freshness, hreflang coverage and image sitemaps.
• Content inventory. Export every page, product, post and category of a site to CSV or Excel in seconds.
• Website migrations. Snapshot all URLs before a redesign or domain move, then check after launch that every old URL redirects (use the status check).
• Change monitoring. Schedule the Actor daily with "Changed since: 1 day" to get only new or updated pages, such as competitor product launches, new blog posts or news articles.
• Feeding AI / RAG pipelines. Get a complete, deduplicated list of a site's URLs, then pass it to a crawler such as Website Content Crawler to build knowledge bases or train chatbots.
• Crawler seeding. Start large crawls with the full URL list instead of following links.
• Competitive research. See how many products, categories or articles a competitor has and how often they are updated.

How to use it

1. Enter one or more websites (example.com) or sitemap URLs (https://example.com/sitemap_index.xml). You can also upload a file with thousands of domains.
2. Optionally set filters ("Changed since", include/exclude regex) and limits.
3. Optionally turn on Check HTTP status of each URL for an SEO audit.
4. Click Start. Download the results as JSON, CSV, Excel or HTML, or get them through the Apify API or integrations (Google Sheets, Zapier, Make, n8n, Slack, webhooks).

Input example

{
    "startUrls": [
        { "url": "https://www.allbirds.com" },
        { "url": "wordpress.org/news" },
        { "url": "https://www.nytimes.com/sitemaps/new/news.xml.gz" }
    ],
    "modifiedSince": "7 days",
    "excludePatterns": ["/tag/", "/author/"],
    "maxUrlsPerSite": 10000,
    "checkStatus": false
}

Output

Each URL is one row in the dataset. Here is a real row from a Shopify store:

{
    "url": "https://www.allbirds.com/products/mens-wool-runners",
    "lastmod": "2026-10-04T09:09:12.000Z",
    "changefreq": "daily",
    "priority": null,
    "title": null,
    "sitemapUrl": "https://www.allbirds.com/sitemap_products_1.xml?from=1878193471557&to=7369944137808",
    "sitemapType": "urlset",
    "discoveredVia": "sitemap-index",
    "site": "https://www.allbirds.com",
    "imageCount": 1,
    "videoCount": 0,
    "hreflangCount": 0,
    "images": [
        {
            "loc": "https://cdn.shopify.com/s/files/1/1104/4168/files/Allbirds_WL_RN_SF_PDP_Natural_Grey_LAT.png?v=1751143404",
            "title": "Men's Wool Runner - Natural Grey (Light Grey Sole)"
        }
    ],
    "videos": [],
    "news": null,
    "alternates": []
}

With Check HTTP status on, each row also gets:

"statusCode": 301, "finalUrl": "https://example.com/new-page/", "finalStatusCode": 200, "redirectCount": 1, "checkMethod": "HEAD", "statusError": null

Rows from a Google News sitemap carry the publication data (real row from theguardian.com):

"title": "NRL grand final 2026: Sydney Roosters v Newcastle Knights – live",
"news": {
    "publicationName": "The Guardian",
    "language": "en",
    "publicationDate": "2026-10-04T09:13:45Z",
    "title": "NRL grand final 2026: Sydney Roosters v Newcastle Knights – live",
    "keywords": "NRL, Sydney Roosters, Newcastle Knights, Rugby league, Australia sport, Sport"
}

Hreflang alternates (real row from spotify.com, 293 alternates) look like "alternates": [{ "hreflang": "en-US", "href": "https://www.spotify.com/us/about-us/contact/" }, { "hreflang": "ca-AD", "href": "https://www.spotify.com/ad-ca/about-us/contact/" }, ...].

The dataset has ready-made views: All URLs, SEO status check, Google News, Images & videos and Hreflang alternates.

Run summary (key-value store, SUMMARY)

Each run also saves a SUMMARY record with a report for every site:

• status: ok, partial, no-sitemap, unreachable, sitemap-error or empty
• how the sitemaps were found
• every sitemap file read, with its type, URL count, HTTP status and any errors or recovered problems
• totals: URLs found, output, duplicates, filtered, broken, redirected

You can see exactly what happened, even for sites that failed.

How much does it cost?

The Actor uses pay-per-event pricing, so you pay only for results:

Event	Price (FREE plan)
Actor start	$0.003 per run
URL extracted	$0.50 per 1,000 URLs
URL status checked (optional)	$1.00 per 1,000 URLs

Paid Apify plans get lower prices. Filtered and duplicate URLs are free, and a site with no sitemap costs nothing beyond the start event. Set a maximum cost per run in the run options and the Actor stops cleanly when it reaches it.

Examples: a 2,500-page store costs about $1.25. A daily "what changed today" check on a news site costs a few cents.

FAQ

Do I need the sitemap URL? No. Just enter the domain and the Actor finds the sitemaps itself. If you already know the sitemap URL, you can enter it directly.

What if the website has no sitemap? The Actor tries robots.txt, about 20 common paths, homepage links and RSS/Atom feeds. If it finds nothing, the site is marked no-sitemap in the summary and the run still succeeds. To discover pages on sites without sitemaps, use a crawler such as Website Content Crawler.

Why are some URLs missing compared to the live site? A sitemap contains what the site owner chose to list. Feeds usually include only the latest 10–50 items.

Does it handle huge sites? Yes. It follows indexes with thousands of child sitemaps, handles files up to 200 MB, and has safety limits ("Max sitemap files per site", "Max URLs"). For very large sites, use "Changed since" and "Skip unchanged child sitemaps".

Can I monitor changes over time? Yes. Create a schedule with "Changed since: 1 day" (or 1 week) and connect a Google Sheets, Slack or webhook integration.

Is it legal? Sitemaps and robots.txt exist so that machines can read them. The Actor reads only these public files, identifies itself with an honest User-Agent and respects robots.txt. The optional status check sends one lightweight request per URL with polite concurrency limits.

Does it need a proxy? Usually not. If a site blocks datacenter IPs, enable Apify Proxy in the input.

Why does the status check use HEAD and then GET? HEAD is the fastest way to check a URL, but many servers and CDNs wrongly answer HEAD with 403, 404 or 405. Every failed HEAD is re-checked with GET (the body is not downloaded), so you don't get false "broken page" alerts.

Limitations

• The Actor does not crawl HTML pages to find links. It reads sitemaps and feeds only.
• Sitemaps that only appear after JavaScript runs are not supported (this is extremely rare).
• lastmod is only as accurate as the website's own sitemap.

Support

Found a site that doesn't work? Open an issue on the Issues tab with the URL. We usually respond within 24 hours, and every reported site becomes a regression test.