Sitemap URL Extractor – Every Page URL from Any Website

Get every URL a website publishes in its XML sitemaps. Auto-discovers sitemaps from robots.txt, follows nested sitemap indexes, handles .gz and .txt sitemaps, returns lastmod, priority, hreflang alternates and image counts, with regex and changed-since filters.

Why this actor

• Reliability vs. the official actor's 17% failure rate; lastmod filter turns it into a cheap 'what changed on competitor sites' monitor.
• Pay only for results: $0.50 per 1,000 urls, no start fee, failed lookups are free.
• Robust: automatic retries with exponential backoff, respects Retry-After, per-item error rows instead of crashed runs.
• Clean, flat JSON/CSV/Excel output ready for Google Sheets, Make, Zapier, n8n, Clay, or the Apify API.

Use cases

• Feed full URL lists into crawlers, LLM/RAG ingestion, or SEO tools
• Monitor competitors' new pages/products (modifiedSince)
• Site migrations & SEO audits
• Count a site's content inventory

How to use Sitemap URL Extractor

1. Configure startUrls, includePattern, excludePattern in the Input tab (or leave the prefilled sample values to test immediately).
2. Optionally set maxItems to cap the number of results and control spend.
3. Click Start and export structured results as JSON, CSV, Excel, or connect directly via API / MCP / n8n / Make.

Input

Field	Type	Description
startUrls	array	A domain (auto-discovers sitemaps via robots.txt) or a direct sitemap.xml / .xml.gz / .txt URL. Nested sitemap indexes are followed.
includePattern	string	e.g. /blog/ or /products/
excludePattern	string	e.g. ?
modifiedSince	string	Only URLs whose
maxSitemaps	integer	Safety cap for huge sites.
maxItems	integer	Default 1000 so runs on huge sites finish fast. Set 0 for no limit (whole sitemap).

Example input:

{
  "startUrls": [
    "https://apify.com"
  ],
  "maxItems": 1000
}

Output

Each dataset item looks like this (real sample from a test run, long fields trimmed):

{
  "url": "https://apify.com/",
  "lastmod": null,
  "changefreq": null,
  "priority": null,
  "imageCount": 0,
  "alternates": null,
  "sitemap": "https://apify.com/sitemap/pages.xml",
  "site": "https://apify.com",
  "scrapedAt": "2026-10-14T05:57:16+00:00"
}

Rows that could not be processed contain an error field and are not charged.

Extracted output fields

Every dataset row includes structured fields ready for spreadsheets, CRMs, and AI pipelines:

• Fields: url, lastmod, changefreq, priority, imageCount, alternates, sitemap, site, scrapedAt

Comparison: Sitemap URL Extractor vs alternatives

Feature	This Actor (shawndunn/sitemap-url-extractor)	Typical alternatives
Pricing model	$0.50 / 1,000 results, $0.00 start fee	$20–$49/mo subscription or per-run start fees
Failed / error lookups	$0.00 (never charged)	Often billed by compute time even on failures
API keys / proxies required	None (works out of the box)	Often requires paid third-party keys or residential proxies
AI Agent & MCP ready	Yes (mcp.apify.com compatible, optional defaults)	Strict required schemas that reject agent calls

Pricing

Pay-per-event: $0.0005 per url (= $0.50 / 1,000). Use Max results to cap spend. Apify's free plan credit covers thousands of results per month.

Integrations & API

Run on a schedule in Apify Console, call via the Apify API / Python & JS clients, or connect to Make, Zapier, n8n, Google Sheets, Slack and webhooks. Works as a tool for AI agents via Apify MCP.

Use with AI agents (MCP)

Sitemap URL Extractor is configured for zero-config AI agent tool use via the Apify MCP Server (no required-field blocking on defaulted inputs).

Streamable HTTP endpoint:

https://mcp.apify.com/?tools=shawndunn/sitemap-url-extractor

Claude Desktop / Cursor / Windsurf mcpServers config:

{
  "mcpServers": {
    "apify-sitemap-url-extractor": {
      "url": "https://mcp.apify.com/?tools=shawndunn/sitemap-url-extractor",
      "headers": {
        "Authorization": "Bearer <YOUR_APIFY_TOKEN>"
      }
    }
  }
}

Direct synchronous API call:

POST https://api.apify.com/v2/acts/shawndunn~sitemap-url-extractor/run-sync-get-dataset-items?token=<YOUR_APIFY_TOKEN>

FAQ

Is it legal? The actor only reads publicly available data from public endpoints, without logins, and does not bypass paywalls or access controls. You are responsible for using the data in line with applicable laws (e.g. GDPR) and the source sites' terms.

Something broke? Open an issue in the Issues tab; we typically respond within 24 hours and ship fixes fast.

How do I get a list of all pages on a website? Run this actor on the homepage. It finds every sitemap the site publishes and returns each URL with lastmod and priority.

Can I track new or updated pages? Yes. Schedule it with modifiedSince set to your last run date to get only new and changed URLs, great for competitor content monitoring.

What if a site has no sitemap? You get a clear error row (not charged). For crawl-based discovery, use a crawler instead.

More tools by shawndunn

• Website Tech Stack Detector — Find what CMS, ecommerce platform, analytics, pixels and frameworks any website uses.
• Domain Intelligence Checker — Bulk domain lookup: WHOIS/RDAP registrar and expiry dates, DNS, MX email provider, SPF, DMARC and SSL certificate expiry.
• Bulk PageSpeed Insights & Core Web Vitals Checker (Lighthouse) — Audit hundreds of URLs with Google PageSpeed Insights: performance, SEO, accessibility and best-practices scores, LCP, CLS, TBT ($3/1k, no start fee).
• Broken Link Checker & Bulk URL Status Checker — Crawl a website and test every internal and external link, or bulk-check a list of URLs (url, statusCode, ok, finalUrl) ($1/1k, no start fee).
• Apple App Store Scraper — Scrape Apple App Store search results, app details, ratings, prices and developer apps in any country.