Sitemap URL Extractor: All Pages from Any Website

Get a complete list of a website's pages in seconds. Enter a domain and this tool finds its XML sitemaps on its own, follows every sitemap index and .gz file, and gives you one clean row per URL with lastmod, changefreq, priority, the sitemap it came from, and any image, video, news and hreflang data. If a site has no sitemap, it falls back to its RSS/Atom feeds and then to a light crawl of internal links, so you always get a page list.

It costs $0.20 per 1,000 URLs. Checking the live HTTP status of each URL is optional and priced separately, so you only pay for it when you need it.

What does Sitemap URL Extractor do?

• Finds sitemaps automatically: reads Sitemap: lines in robots.txt, then checks the usual locations (/sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap/sitemap.xml, /sitemaps/sitemap.xml, /sitemap.txt). This covers WordPress (core, Yoast, Rank Math, Jetpack), Shopify, Wix, Squarespace, Webflow, Next.js and most other platforms. robots.txt often lists only some of a site's sitemaps, so the main locations are always checked too.
• Follows sitemap indexes to any depth and reads gzip-compressed (.gz) sitemaps, plain-text sitemaps and RSS / Atom feeds used as sitemaps.
• Reads Google's sitemap extensions: image sitemaps (image URL, title, caption), video sitemaps (title, description, thumbnail, content and player URLs, duration), news sitemaps (publication, language, publication date, title, keywords) and xhtml:link hreflang alternates.
• Filters while it reads: keep only URLs matching a pattern, drop URLs matching another, keep only pages modified after a date. With a date filter, child sitemaps whose own lastmod is older than the date are skipped entirely, which makes runs on very large sites fast.
• Handles sections: enter https://example.com/blog/ and you get only blog URLs. It also looks for a sitemap inside that folder first (common for WordPress installed in a sub-folder).
• Removes duplicates across sitemaps and sites.
• Optional status check: one light HEAD request per URL (GET if the server refuses HEAD) adds statusCode, finalUrl, redirectChain and contentType, so you can spot 404s, redirects and dead entries in your sitemap.
• Fallback: no sitemap and no feed? It crawls internal links from the home page (up to a page limit you set) and marks those rows source: "crawled", with page title and hreflang links.
• Fast and light: plain HTTP requests, no browser and no proxy. A 1,000-URL site typically finishes in under 10 seconds.

Who uses it?

• SEO specialists and agencies: audit sitemap coverage, find stale lastmod values, check that every listed URL returns 200, and compare hreflang sets.
• Content and marketing teams: get an inventory of every blog post, product or landing page with its last update date.
• Developers and data teams: build a URL list to feed a scraper, a migration script, a search index or a monitoring job.
• AI and RAG builders: collect the exact page list to crawl, or only pages changed since your last sync.
• Competitive research: see how many products, articles or locations a competitor publishes and how often they change.

How to use it

1. Click Try for free (or Start).
2. Under Websites or sitemap URLs, enter one or more domains (example.com), sections (https://example.com/blog/) or direct sitemap / feed URLs.
3. Set Maximum URLs per website. This also caps your cost.
4. Optional: add Only URLs matching / Skip URLs matching patterns, set Only URLs modified after (e.g. 2025-01-01 or 30 days), or turn on Check HTTP status of every URL.
5. Run it and open the URLs table, or export to CSV, Excel, JSON or XML. The key-value store record SUMMARY lists every sitemap file that was read for each site.

Input

Field	Default	Description
startUrls	wordpress.org/news	Domains, home pages, sections or direct sitemap / RSS URLs
maxUrlsPerSite	1000	URLs to output per website (after filters and de-duplication)
stayInStartPath	true	For a section URL such as /blog/, keep only URLs under that folder
includePatterns	empty	Keep only URLs matching one of these regular expressions or * wildcards
excludePatterns	empty	Drop URLs matching any of these
lastmodAfter	empty	Keep URLs with lastmod (or news / feed date) on or after this date; YYYY-MM-DD or 7 days, 3 months
keepUrlsWithoutLastmod	false	With a date filter, also keep URLs that have no date
dedupe	true	Output each URL once
includeExtras	true	Add images, videos, news and hreflang alternates
checkStatus	false	HEAD/GET every URL for its status code and redirects (charged separately)
crawlIfNoSitemap	true	Crawl internal links when no sitemap or feed exists
maxCrawlPages	100	Page limit for that fallback crawl
sameDomainOnly	false	Drop sitemap entries that point to other domains
statusCheckConcurrency	5	Parallel status checks (1-10)
timeoutSecs	20	Timeout per status check

Input example

{
  "startUrls": ["example.com", "https://shop.example.org/", "https://news.example.net/sitemap-news.xml"],
  "maxUrlsPerSite": 50000,
  "excludePatterns": ["/tag/", "/author/"],
  "lastmodAfter": "90 days",
  "checkStatus": true
}

Output example

A real record from the default run (wordpress.org/news):

{
  "url": "https://wordpress.org/news/2003/04/welcome/",
  "lastmod": "2021-06-04T11:57:33Z",
  "changefreq": null,
  "priority": null,
  "source": "sitemap",
  "sitemapUrl": "https://wordpress.org/news/sitemap-1.xml",
  "sitemapType": "urlset",
  "sitemapFoundVia": "sitemap index",
  "site": "https://wordpress.org/news/",
  "imageCount": 0,
  "images": null,
  "videos": null,
  "news": null,
  "alternates": null
}

A Shopify product URL from a test run, showing image sitemap data:

{
  "url": "https://www.allbirds.com/products/mens-wool-runners",
  "lastmod": "2026-11-01T11:12:57-08:00",
  "changefreq": "daily",
  "sitemapUrl": "https://www.allbirds.com/sitemap_products_1.xml?from=1878193471557&to=7369944137808",
  "imageCount": 1,
  "images": [{ "url": "https://cdn.shopify.com/s/files/1/1104/4168/files/Allbirds_WL_RN_SF_PDP_Natural_Grey_LAT.png?v=1751143404", "title": "Men's Wool Runner - Natural Grey (Light Grey Sole)" }]
}

Output fields

Field	Meaning
url	Page URL as listed in the sitemap (or found by the crawl)
lastmod, changefreq, priority	Values from the sitemap, exactly as published (feeds: publication date in ISO format)
source	sitemap, feed (RSS/Atom) or crawled
sitemapUrl, sitemapType	The file the URL came from and its type (urlset, rss, atom, text)
sitemapFoundVia	robots.txt, common path, sitemap index, home page <link>, input or crawl (no sitemap found)
site	The input this row belongs to
images, imageCount	Image sitemap entries (url, title, caption)
videos	Video sitemap entries
news	News sitemap data (publicationName, language, publicationDate, title, keywords)
alternates	hreflang alternates (hreflang, url)
title	Feed item title or crawled page title
statusCode, finalUrl, redirectChain, contentType, statusNote	Only with status checking on

How much does it cost?

You pay per result with Apify's pay-per-event pricing:

• URL extracted: $0.20 per 1,000 URLs ($0.0002 each)
• URL status checked (optional): $0.25 per 1,000 URLs, only when Check HTTP status is on
• A tiny start fee of $0.00005 per run

Worked examples:

• 1,000-page company site: $0.20
• 25,000-product Shopify store: $5.00; with status checks $11.25
• Weekly "what changed in the last 7 days" run on a news site returning 3,000 URLs: $0.60

Volume discounts apply automatically on higher Apify plans (Bronze, Silver, Gold and above). Apify's free monthly credit covers tens of thousands of URLs. Set Maximum URLs per website or a maximum cost per run to stay within budget; the run stops cleanly when the limit is reached.

Use it via API, Make, Zapier, n8n or an AI agent

Every run can be started through the Apify API and returns data as JSON, CSV or Excel:

curl -X POST "https://api.apify.com/v2/acts/plotpointtech~sitemap-url-extractor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["example.com"], "maxUrlsPerSite": 5000}'

You can also schedule it in Apify Console, or connect it to Make, Zapier, n8n, Google Sheets, Slack and other tools through Apify integrations. AI agents (Claude, ChatGPT, Cursor and others) can call it through the Apify MCP server, for example to fetch a site's page list before summarising or crawling it.

FAQ

Is it legal to extract sitemap URLs? Sitemaps and robots.txt are published by website owners specifically so that automated tools can discover their pages. This tool only reads public files, never logs in, does not use proxies or bypass blocking, and respects robots.txt for the fallback crawl and status checks. You are responsible for how you use the data and for following each website's terms.

Why did I get fewer URLs than I expected? Check the SUMMARY record in the key-value store and the run log: they list every sitemap read, URLs filtered out, duplicates removed and any sitemap that returned an error. Some sites block automated requests (HTTP 403 or 429); those sitemaps cannot be read.

Does it handle huge sites? Yes. Sitemaps are parsed as a stream, so 50,000-URL files and indexes with thousands of child sitemaps work. Use lastmodAfter to skip unchanged child sitemaps and finish much faster.

What happens with the same URL in two sitemaps? With de-duplication on (the default), the first occurrence is kept. Turn it off to get every occurrence, for example to audit which sitemaps list a page.

Does it render JavaScript? No. Sitemaps and feeds do not need it. The fallback crawl reads links in the HTML, which is how search engines first see a page.

Why did a run take longer than usual? Some websites ask crawlers to wait between requests (Crawl-delay in robots.txt). The tool honours that delay, up to 10 seconds per request.

Related tools from Plot Point Technologies

• SEO Audit Crawler: titles, meta descriptions, H1s, canonicals and status codes for every page in a sitemap.
• Broken Link Checker: crawl a site and find every 404, redirect and dead image.
• Website to Markdown Crawler: turn the pages you found into clean Markdown for AI and RAG.
• Website Tech Stack Detector: CMS, analytics and other technologies behind a site.
• PageSpeed Insights & Core Web Vitals Checker: Lighthouse scores for many URLs.

Plot Point Technologies is not affiliated with WordPress.org, Shopify, Allbirds, any website you extract, or Apify.

---

If this tool saved you time, we would be grateful for a short review on this page; it helps other people find it. If something did not work as expected, please open an issue on the Issues tab and we will look into it quickly.