Sitemap URL Extractor: Get All URLs of Any Website

Extract every URL from a website's sitemap in seconds. Enter a domain and the Actor finds its sitemaps automatically (robots.txt + common locations), follows sitemap indexes of any depth, reads gzipped .xml.gz, plain-text, RSS and Atom sitemaps, and returns a clean list of URLs with lastmod, changefreq, priority, hreflang alternates, image and Google News data.

Schedule it with a monitoring key and it tells you which pages are new since the last run: perfect for tracking competitor blogs, product launches and content changes.

No browser, no crawling of HTML pages: it only reads the sitemaps the site publishes for search engines, so it is fast and cheap (hundreds of thousands of URLs per minute).

What can this sitemap extractor do?

• 🔎 Auto-discovers sitemaps from robots.txt (Sitemap: lines), falling back to /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap.xml.gz and /sitemap.txt.
• 🗂️ Follows sitemap index files recursively (WordPress, Yoast, Rank Math, Shopify, Wix, Squarespace, Next.js, Docusaurus and custom setups).
• 📦 Reads gzip-compressed sitemaps, text sitemaps (one URL per line), RSS 2.0 and Atom feeds.
• 🌍 Extracts hreflang alternates (xhtml:link) so you can audit international SEO.
• 🖼️ Counts and optionally lists image sitemap URLs; reads Google News sitemap title, publication and date.
• 🧹 Deduplicates URLs across all sitemaps of a site and filters by URL pattern (contains, wildcard or regex) and lastmod date ("modified in the last 7 days").
• 🆕 New-URL monitoring: flags every URL with isNew versus the previous run and reports new and removed URL counts.
• 🧾 A per-site SITES report: which sitemaps were found, URL counts per sitemap and a clear error for every sitemap that failed (404, blocked, not XML, HTML page...).

Use cases

• SEO audits & migrations: get the full list of indexable URLs, compare old vs new sitemaps, check hreflang coverage and lastmod freshness.
• Competitor monitoring: get notified when a competitor publishes new blog posts, landing pages, products or docs.
• Feed a crawler: use the URL list as start URLs for Website Content Crawler, our Bulk PageSpeed & Core Web Vitals Auditor or our Website Screenshot API.
• Content inventories: count pages per section (/blog/, /products/, /docs/) and see how often they change.
• News tracking: read Google News sitemaps for the latest articles from publishers.
• AI / RAG pipelines: discover all documentation pages of a product before indexing them.

How to use it

1. Enter one or more websites (https://example.com, or just example.com) or paste sitemap URLs directly.
2. Optionally add filters (e.g. /blog/), a Modified after date (7 days) or a Max URLs per site limit.
3. Click Start. Download the results as JSON, CSV, Excel or HTML, or get them via the API.

To monitor a site for new pages: set Monitoring key (any name, e.g. competitor-blogs), turn on Output only new URLs, and create a Schedule (e.g. daily). The first run records the baseline; each following run outputs only the new URLs. Connect Slack, email, Google Sheets, Zapier, Make or a webhook to get alerts.

Input example

{
  "startUrls": ["https://crawlee.dev", "https://www.theguardian.com/sitemaps/news.xml"],
  "includePatterns": ["/docs/"],
  "modifiedAfter": "30 days",
  "maxUrlsPerSite": 5000
}

Field	What it does	Default
startUrls	Websites (auto-discovery) or sitemap URLs	["https://crawlee.dev"]
maxUrlsPerSite	Cap on saved URLs per site, 0 = all	0
includePatterns / excludePatterns	Contains text, * wildcard or /regex/	–
modifiedAfter	YYYY-MM-DD or 7 days, 24 hours, 2 weeks	–
sameHostOnly	Drop URLs on other hosts	false
monitorStateKey / onlyNewUrls	New-URL monitoring	– / false
includeAlternates / includeImages / includeNews	Extra fields	true / false / true
discoverFromRobotsTxt / tryCommonLocations	Sitemap discovery	true / true
proxyConfiguration	Optional Apify Proxy for sites that block	off

Output example

One row per unique URL:

{
  "url": "https://crawlee.dev/js/docs/introduction",
  "site": "https://crawlee.dev",
  "sitemapUrl": "https://crawlee.dev/sitemap.xml",
  "sitemapType": "urlset",
  "lastmod": null,
  "lastmodRaw": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "alternates": [],
  "imageCount": 0,
  "discoveredAt": "2026-10-19T15:40:12.345Z"
}

With a monitoring key every row also has "isNew": true|false. News sitemaps add newsTitle, newsPublicationName, newsPublicationDate and newsLanguage. The SITES record in the run's key-value store lists every sitemap file found, its type and URL count, skipped sitemaps, and the new/removed URL counts per site.

How much does it cost to extract sitemap URLs?

You pay $0.0002 per unique URL saved ($0.20 per 1,000). Failed sitemaps, filtered-out URLs and unchanged URLs in "only new" mode are free. No start fee.

Job	URLs	Cost
Small blog	300	$0.06
1,000 URLs	1,000	$0.20
Mid-size e-commerce site	25,000	$5
100,000 URLs	100,000	$20
Daily new-page monitor of 10 competitor sites (~50 new URLs/day)	~1,500/month	~$0.30/month

• Apify's free plan includes $5 of monthly credit, i.e. about 25,000 URLs a month for free.
• Paid Apify plans get automatic discounts: Silver -10%, Gold and above -20%.
• Use Max URLs per site or Maximum cost per run to cap spend.

Sitemap extractor vs. the alternatives

	This Actor	Online "sitemap checker" pages	Full website crawler	Writing a script
Finds sitemaps via robots.txt + common paths	✅	sometimes	–	DIY
Nested sitemap indexes, .xml.gz, text, RSS/Atom	✅	partial	–	DIY
lastmod, hreflang, image and Google News data	✅	rarely	❌	DIY
Many domains per run	✅	❌ one at a time	✅	✅
New / removed URL detection between runs	✅	❌	❌	DIY state
Speed / cost	100k+ URLs per minute, $0.20 per 1,000	free, capped	slow, renders pages	your time

Use it as an API (Python, JavaScript, cURL, MCP)

Every Apify Actor is also a REST API, so you can call this sitemap extractor from your own code, a no-code tool or an AI agent.

Python

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("datawright-labs/sitemap-url-extractor").call(run_input={"startUrls": ["https://crawlee.dev"], "maxUrlsPerSite": 1000})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

JavaScript / Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('datawright-labs/sitemap-url-extractor').call({"startUrls": ["https://crawlee.dev"], "maxUrlsPerSite": 1000});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

cURL (run and get results in one request)

curl -X POST "https://api.apify.com/v2/acts/datawright-labs~sitemap-url-extractor/run-sync-get-dataset-items?token=<YOUR_APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://crawlee.dev"], "maxUrlsPerSite": 1000}'

AI agents (MCP)

Add https://mcp.apify.com/?tools=datawright-labs/sitemap-url-extractor to Claude, Cursor or any MCP client and the agent can run it as a tool.

Results can also go straight to Google Sheets, Slack, Make, Zapier, n8n, Airbyte or a webhook via Apify integrations, and you can run it on a Schedule.

FAQ

The site has no sitemap. What happens?

The site is reported in the SITES record with the error "No sitemap found", and you pay nothing. If you know the sitemap address (some sites use custom paths), paste it directly.

A site returns 403 / 429. Some sites block datacenter traffic. Enable Apify Proxy in the Proxy section and run again.

Does it crawl pages that are not in the sitemap?

No. It reads only sitemaps, which is what makes it fast and cheap. To discover pages by following links, use a crawler such as Website Content Crawler on the URLs this Actor returns.

How many URLs can it handle?

Large sites are fine: a test run extracted ~766,000 URLs from apify.com's 18 sitemaps in under 10 seconds of download time. Use 2-4 GB memory for multi-million-URL sites.

Is it legal?

Sitemaps are public files that site owners publish specifically so that automated tools can discover their pages. The Actor collects no personal data. You are responsible for how you use the URLs and the pages behind them.

Something is missing? Open an issue on the Issues tab; we usually reply within a day.

How do I find all pages of a website?

Most sites list every indexable page in their XML sitemap. Enter the domain and this Actor finds and reads all of its sitemaps, returning every URL with its last-modified date. For pages not in the sitemap, run a crawler on top.

How do I extract URLs from sitemap.xml to CSV or Excel?

Run the Actor with the sitemap or domain, then download the dataset as CSV/Excel, or send it to Google Sheets with the integration.

How can I monitor a competitor's new pages or blog posts?

Set a monitoring key, turn on "only new URLs", and schedule the Actor daily. Each run returns just the pages added since the last run.

More tools from Datawright Labs

• Bulk PageSpeed & Core Web Vitals Checker: audit PageSpeed / Lighthouse scores for many URLs.
• Website Screenshot API: bulk full-page screenshots and PDFs.
• Google Autocomplete Keyword Scraper: long-tail keyword ideas from Google, YouTube and Bing.
• Greenhouse, Lever & Ashby Jobs Scraper: normalized job feeds from 5 ATS platforms.
• Broken Link Checker: find broken links, images and redirects across a whole site.
• App Store Top Charts & Rank Tracker: daily top free/paid/grossing app rankings with rank changes.

Tip: pass the URLs to the Bulk PageSpeed & Core Web Vitals Checker or the Website Screenshot API to audit or archive a whole site.

Made by Datawright Labs: reliable, fairly priced web data tools. Questions or a missing feature? Open an issue on the Issues tab; we usually reply within a day.