Sitemap URL Extractor: Get All URLs of a Website from Its Sitemaps

Paste a domain and get every page URL the site publishes in its XML sitemaps, as clean JSON, CSV or Excel. The Actor finds the sitemaps for you (robots.txt Sitemap: lines, then /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and other standard locations), follows sitemap indexes to any depth, opens .xml.gz and plain-text sitemaps, and returns each URL with its lastmod date, changefreq, priority, hreflang alternates, image/video counts and Google News title/date. Turn on HTTP status check to find broken (404) or redirected pages listed in a sitemap. The price is $0.40 per 1,000 URLs.

What you can use it for

• SEO audits. Count indexable pages, spot 404s and redirects that should not be in the sitemap, check hreflang coverage.
• Crawl seeding. Feed the URL list into a scraper (Website Content Crawler, your own Crawlee code) instead of crawling blind.
• Content and competitor monitoring. Use lastmodAfter to list only pages published or updated since a date: new products, new blog posts, new docs.
• Migrations. Export the full URL inventory before a redesign and build redirect maps.
• News monitoring. Google News sitemaps give the headline and publication time for each article.

Why this sitemap scraper?

• Reliable by design. Status-aware retries with backoff and Retry-After, a regex fallback for malformed XML, gzip detection by content (not only by file name), and a per-host concurrency limit. Sites that block us or have no sitemap are listed in the FAILED_SITEMAPS and SITES_WITHOUT_SITEMAP records instead of failing the whole run.
• Fast and cheap. It reads only sitemap files, not pages. In our tests 4,800 URLs from crawlee.dev took about 3 seconds.
• Fair billing. You pay only for unique URLs saved. Duplicates and URLs removed by your filters are free. The optional status check is billed only for URLs that actually returned a status.
• Polite. Status checks respect robots.txt and use few parallel requests per host.

Input

Field	Description
startUrls	Domains (example.com), homepages, or direct sitemap URLs (.xml, .xml.gz, .txt).
maxUrls	Total cap across all sites (default 10,000).
maxUrlsPerSite	Optional cap per site (0 = none).
includePatterns / excludePatterns	Keep or drop URLs by substring, glob (*/products/*) or regex (re:^https://[^/]+/p/\d+).
lastmodAfter	ISO date. Only URLs whose <lastmod> is on or after it.
checkHttpStatus	Adds httpStatus and redirectTo for each URL (HEAD, GET fallback).
sameDomainOnly, maxSitemapsPerSite, perHostConcurrency	Advanced controls.

Example:

{
  "startUrls": ["https://blog.apify.com", "https://wordpress.org/news/"],
  "includePatterns": ["*scrap*"],
  "lastmodAfter": "2026-01-01",
  "maxUrls": 5000
}

Output example

{
  "url": "https://blog.apify.com/best-price-scraping-tools/",
  "lastmod": "2026-10-07T09:14:00Z",
  "lastmodRaw": "2026-10-07T09:14:00.000Z",
  "changefreq": null,
  "priority": null,
  "imageCount": 1,
  "videoCount": 0,
  "newsTitle": null,
  "newsPublicationDate": null,
  "alternates": [],
  "sitemapUrl": "https://blog.apify.com/sitemap-posts.xml",
  "site": "https://blog.apify.com",
  "discoveredVia": "robots.txt",
  "httpStatus": 200,
  "extractedAt": "2026-10-08T06:10:00Z"
}

httpStatus, redirectTo and statusNote appear only when checkHttpStatus is on. alternates lists hreflang links as { "hreflang": "de", "href": "..." }.

Pricing

Event	Price
URL extracted (unique URL saved)	$0.40 per 1,000
URL status checked (optional)	$1.00 per 1,000

Set Maximum cost per run in the run options and the Actor stops cleanly when it is reached. You are never charged for duplicates, filtered URLs, failed sitemaps or sites without a sitemap.

Run it from code: API, Python, JavaScript or cURL

Every Apify Actor is also a REST API. Get your token in Apify Console under Settings → API & Integrations, then:

cURL (runs the Actor and returns the URLs in one call):

curl -X POST "https://api.apify.com/v2/acts/saadhazz~sitemap-url-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://crawlee.dev"], "maxUrls": 1000}'

Python (pip install apify-client):

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("saadhazz/sitemap-url-extractor").call(run_input={"startUrls": ["https://crawlee.dev"], "maxUrls": 1000})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

JavaScript / Node.js (npm i apify-client):

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('saadhazz/sitemap-url-extractor').call({"startUrls": ["https://crawlee.dev"], "maxUrls": 1000});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Use it with AI agents (MCP), Make, Zapier, n8n and Google Sheets

• AI agents and LLM apps: connect Claude, Cursor, ChatGPT or any MCP client to https://mcp.apify.com/?tools=saadhazz/sitemap-url-extractor and the agent can call this Actor as a tool.
• No-code: use the official Apify integrations for Make, Zapier and n8n, or export the dataset straight to Google Sheets, CSV, Excel or JSON.
• Scheduling and webhooks: create a Schedule in Apify Console (e.g. every morning) and add a webhook or Slack/email integration to get notified when new data is ready.

FAQ

The site has no sitemap. What happens? It is listed in the SITES_WITHOUT_SITEMAP record and costs nothing. This Actor does not crawl pages to discover links.

Does it handle huge sites? Yes. Sitemap indexes are followed to any depth (up to maxSitemapsPerSite files per site, default 500, each up to 50 MB uncompressed).

Why are some URLs missing a lastmod? The site did not publish one. With lastmodAfter set, such URLs are dropped.

Some sitemaps return 403. A few sites block automated clients for their sitemaps. Those files are listed in FAILED_SITEMAPS.

Legal note

The Actor reads only sitemap files and robots.txt, which websites publish specifically for automated clients, and (optionally) the HTTP status of listed URLs while respecting robots.txt. It collects no personal data. You are responsible for how you use the URLs and for complying with each website's terms.

Changelog

• 1.0 (2026-10-08): first release.

More data APIs from the same developer

• Job Search API: Career Site Jobs: search 400k+ jobs from 13,000+ company career sites (Greenhouse, Lever, Ashby, Workable…) by title, country and remote, with new-job alerts. $2 / 1,000 jobs.
• ATS Jobs API: every open job from a list of companies on Greenhouse, Lever, Ashby, SmartRecruiters or Recruitee, in one schema. $1 / 1,000 jobs.
• Remote Jobs API: remote and work-from-home jobs from Remote OK, Himalayas, Jobicy and Arbeitnow, deduplicated. $1 / 1,000 jobs.
• Bulk WHOIS & RDAP Lookup: domain expiry dates, registrar, age, status, nameservers, MX/SPF/DMARC for thousands of domains. $1 / 1,000 domains.
• PDF Text Extractor: PDF URLs to clean page-by-page text, Markdown tables and metadata for RAG/LLM pipelines.
• SEC EDGAR Filings Monitor: new 8-K, 10-Q, 10-K and Form 4 filings for a list of tickers from the official SEC API.

Found a bug or need a field? Open an issue on the Issues tab; Issues are checked daily.