Sitemap URL Extractor – Every Page URL from Any Website

Get every URL a website publishes in its XML sitemaps. Auto-discovers sitemaps from robots.txt, follows nested sitemap indexes, handles .gz and .txt sitemaps, returns lastmod, priority, hreflang alternates and image counts, with regex and changed-since filters.

Why this actor

• Reliability vs. the official actor's 17% failure rate; lastmod filter turns it into a cheap 'what changed on competitor sites' monitor.
• Pay only for results: $0.50 per 1,000 urls, no start fee, failed lookups are free.
• Robust: automatic retries with exponential backoff, respects Retry-After, per-item error rows instead of crashed runs.
• Clean, flat JSON/CSV/Excel output ready for Google Sheets, Make, Zapier, n8n, Clay, or the Apify API.

Use cases

• Feed full URL lists into crawlers, LLM/RAG ingestion, or SEO tools
• Monitor competitors' new pages/products (modifiedSince)
• Site migrations & SEO audits
• Count a site's content inventory

How to use Sitemap URL Extractor

1. Enter a homepage (e.g. https://apify.com) or a direct sitemap URL.
2. Optionally add include/exclude regex (e.g. /blog/) or modifiedSince to get only changed pages.
3. Click Start and download the full URL list, or feed it into another actor as start URLs.

Input

Field	Type	Description
startUrls	array	A domain (auto-discovers sitemaps via robots.txt) or a direct sitemap.xml / .xml.gz / .txt URL. Nested sitemap indexes are followed.
includePattern	string	e.g. /blog/ or /products/
excludePattern	string	e.g. ?
modifiedSince	string	Only URLs whose
maxSitemaps	integer	Safety cap for huge sites.
maxItems	integer	Default 1000 so runs on huge sites finish fast. Set 0 for no limit (whole sitemap).

Example input:

{
  "startUrls": [
    "https://apify.com"
  ],
  "maxItems": 1000
}

Output

Each dataset item looks like this (real sample from a test run, long fields trimmed):

{
  "url": "https://apify.com/",
  "lastmod": null,
  "changefreq": null,
  "priority": null,
  "imageCount": 0,
  "alternates": null,
  "sitemap": "https://apify.com/sitemap/pages.xml",
  "site": "https://apify.com",
  "scrapedAt": "2026-10-14T05:57:16+00:00"
}

Rows that could not be processed contain an error field and are not charged.

Comparison: Sitemap URL Extractor vs alternatives

	This actor	Typical alternatives
Sitemap discovery	robots.txt + common paths, nested indexes, .gz	Often one sitemap URL only
Changed-since filter	Yes (lastmod)	Rare
Scale	100k+ URLs per run, no browser	Desktop tools limit free tier to 500 URLs
Pricing	$0.50 / 1,000 URLs	Subscription

Pricing

Pay-per-event: $0.0005 per url (= $0.50 / 1,000). Use Max results to cap spend. Apify's free plan credit covers thousands of results per month.

Integrations & API

Run on a schedule in Apify Console, call via the Apify API / Python & JS clients, or connect to Make, Zapier, n8n, Google Sheets, Slack and webhooks. Works as a tool for AI agents via Apify MCP.

Use with AI agents (MCP)

Sitemap URL Extractor is a ready-made tool for AI agents (sitemap.xml parser and website URL list extractor). Connect it to Claude, ChatGPT, Cursor, VS Code or any MCP client with the Apify MCP server:

https://mcp.apify.com/?tools=shawndunn/sitemap-url-extractor

Agents can also call it directly via the Apify API (POST https://api.apify.com/v2/acts/shawndunn~sitemap-url-extractor/run-sync-get-dataset-items) and pay per result, with no subscription.

FAQ

Is it legal? The actor only reads publicly available data from public endpoints, without logins, and does not bypass paywalls or access controls. You are responsible for using the data in line with applicable laws (e.g. GDPR) and the source sites' terms.

Something broke? Open an issue in the Issues tab; we typically respond within 24 hours and ship fixes fast.

How do I get a list of all pages on a website? Run this actor on the homepage. It finds every sitemap the site publishes and returns each URL with lastmod and priority.

Can I track new or updated pages? Yes. Schedule it with modifiedSince set to your last run date to get only new and changed URLs, great for competitor content monitoring.

What if a site has no sitemap? You get a clear error row (not charged). For crawl-based discovery, use a crawler instead.

More tools by shawndunn

• ATS Jobs Scraper - Scrape jobs from Greenhouse, Lever, Ashby, Workable, SmartRecruiters & Recruitee boards.
• Website Tech Stack Detector - Find what CMS, ecommerce platform, analytics, pixels and frameworks any website uses.
• Domain Intelligence Checker - Bulk domain lookup: WHOIS/RDAP registrar and expiry dates, DNS, MX email provider, SPF, DMARC and SSL certificate expiry.
• Apple App Store Scraper - Scrape Apple App Store search results, app details, ratings, prices and developer apps in any country.