Sitemap Extractor: All Website URLs + New/Removed Pages

Sitemap extractor: get every URL from a website's sitemap.xml in one table, then schedule it to track new, updated and removed pages.

Get every URL a website publishes in its XML sitemaps in one clean table. Then schedule it to see what changed: new pages, updated pages and removed pages since the last run.

• Automatic discovery. Sitemaps are found in robots.txt and at common locations (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, and gzip versions). You can also pass sitemap URLs directly.
• Handles real-world sitemaps. Nested sitemap indexes, .xml.gz, plain-text sitemaps, RSS used as a sitemap, broken or truncated XML (recovered), and files with 50,000+ URLs. Shopify, WordPress/Yoast, Next.js and custom CMSs all work.
• Rich fields. lastmod, changefreq, priority, image URLs, hreflang alternates, Google News tags, video count, and the sitemap each URL came from.
• Change tracking. Each URL is marked new, updated (lastmod changed), unchanged or removed, with the previous lastmod. Use Only changes to monitor competitors cheaply.
• Optional HTTP status check. Find 404s and redirects in your own sitemap, a common SEO problem.
• Section scoping. Enter example.com/blog/ to inventory just that section. You can also use include/exclude regex and a "modified after" date.

Use cases

• SEO audits. Get a full URL inventory, find stale lastmod dates, missing hreflang, and 404s or redirects listed in the sitemap.
• Competitor monitoring. Schedule daily runs in Only changes mode to see every new product, landing page or blog post a competitor publishes. Connect Slack or email.
• E-commerce intelligence. Track product launches and discontinued products (removed URLs) on Shopify stores.
• Crawl seeding. Feed URLs into your scraper or into Documents to AI-Ready Markdown for RAG.

Input example

{
  "startUrls": ["https://www.allbirds.com", "https://example.com/blog/"],
  "outputMode": "changes",
  "compareWithPrevious": true,
  "stateName": "competitors",
  "includePatterns": ["/products/"]
}

Output

URL rows:

{
  "type": "url", "site": "https://www.allbirds.com",
  "url": "https://www.allbirds.com/products/mens-wool-runners",
  "changeStatus": "updated", "lastmod": "2026-10-11T10:36:01-07:00", "previousLastmod": "2026-10-04T08:00:00-07:00",
  "changefreq": "daily", "imageCount": 1, "images": ["https://cdn.shopify.com/..."], "alternates": [],
  "sitemapUrl": "https://www.allbirds.com/sitemap_products_1.xml?from=...", "httpStatus": 200
}

Each site also gets a free site summary row with sitemap files found, URL count, new/updated/unchanged/removed counts and errors.

Pricing (pay per event)

Event	Price
Site scanned	$0.003
URL listed (each output row)	$0.0003, i.e. $0.30 per 1,000 URLs
URL status checked (optional)	$0.001
Sites without a sitemap	free

For example, a daily changes-only monitor of 10 competitor sites with about 20 changes a day costs about $0.04 a day. If your spending limit stops a run early, the snapshot isn't advanced, so the undelivered changes are reported again in the next run.

FAQ

Where is the snapshot stored? In your account, in a named key-value store (sitemap-inventory-<stateName>), compressed.

What if the site has more URLs than my cap? You get the first N URLs. Removal detection is turned off for that site in that run, so you won't get false "removed" alerts.

Does it crawl pages? No, it only reads sitemaps (plus optional HEAD status checks). That makes it fast, cheap and polite.

---

Built by PewterDesk (pewterdesk.pages.dev). More tools: apify.com/pewterdesk.