Sitemap URL Extractor: every page URL, lastmod & new pages from any website

🗺️ What is the Sitemap URL Extractor?

Get the complete list of pages of any website from its XML sitemaps, in seconds. Enter a domain and the Actor finds the sitemaps automatically (robots.txt plus the usual locations), follows sitemap indexes recursively, reads gzip, plain-text and RSS/Atom sitemaps, and returns one clean row per URL with lastmod, changefreq, priority, hreflang alternates, images and Google News data.

Schedule it to monitor sites for new pages: it remembers what it saw and flags isNew, or outputs only new URLs since the last run.

• ✅ Auto-discovery from robots.txt Sitemap: lines, /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml ...
• ✅ Sitemap indexes (any depth), .xml.gz, .txt and RSS/Atom feed sitemaps
• ✅ lastmod, changefreq, priority, hreflang alternates, images, newsTitle / newsPublishedAt
• ✅ Filters: include / exclude regex, modified since, same host only
• ✅ New-URL tracking and only new URLs mode for monitoring
• ✅ Handles big sites: hundreds of sitemap files, hundreds of thousands of URLs, streamed parsing
• ✅ Respects robots.txt, polite rate limiting, no browser and no proxy needed
• ✅ Error report per site in RUN_SUMMARY (which sitemaps failed and why)

🎯 What can I use it for?

• SEO audits: list every indexable page, find stale pages by lastmod, check hreflang coverage.
• Competitor monitoring: get alerted when a competitor publishes new pages, products, landing pages or blog posts.
• Crawl seeding: feed the URL list into a scraper, Website Content Crawler, RAG pipeline or a link checker.
• Content inventory & migrations: export every URL to CSV/Excel before a redesign or CMS migration.
• News & blog tracking: Google News sitemaps give title and publication time of fresh articles.
• AI agents: a fast "what pages does this site have?" tool via the Apify API or MCP.

⬇️ Input

Field	Description
startUrls (required)	Websites (apify.com) or sitemap URLs (https://site.com/sitemap_index.xml).
maxUrls	Total cap (default 10,000; 0 = no limit). Also caps your cost.
maxUrlsPerSite	Per-site cap (0 = none).
includePatterns / excludePatterns	Regular expressions on the URL, e.g. /blog/, /tag/.
lastmodSince	Only URLs modified since 2026-10-01 or 7 days.
keepUrlsWithoutLastmod	With a date filter, keep URLs that have no lastmod (default on).
sameHostOnly	Drop URLs on other hosts.
includeExtensions	hreflang, images and news fields (default on).
trackNewUrls / onlyNewUrls	Track URLs between runs (default on) / output only new ones.
stateStoreName	Named store for tracking; one per monitor.
maxSitemapsPerSite	Safety limit for huge indexes (default 1,000).
respectRobotsTxt	Skip sitemap files disallowed by robots.txt (default on).

Example input

{
  "startUrls": ["apify.com", "https://blog.cloudflare.com/sitemap.xml"],
  "includePatterns": ["/blog/"],
  "lastmodSince": "30 days",
  "maxUrls": 5000
}

⬆️ Output

{
  "url": "https://blog.cloudflare.com/zh-cn/agentic-web",
  "lastmod": "2026-10-08T07:40:32.088000Z",
  "changefreq": null,
  "priority": null,
  "site": "https://blog.cloudflare.com",
  "sitemapUrl": "https://blog.cloudflare.com/sitemap-posts.xml",
  "alternates": [{"hreflang": "pt-br", "url": "https://blog.cloudflare.com/pt-br/agentic-web"}],
  "images": [],
  "imageCount": 0,
  "newsTitle": null,
  "newsPublishedAt": null,
  "isNew": true,
  "firstSeenAt": "2026-10-08T08:10:00Z",
  "scrapedAt": "2026-10-08T08:10:00Z"
}

RUN_SUMMARY (default key-value store) lists, per site: how sitemaps were discovered, sitemap files read, URLs found / output, new URLs and errors.

💰 How much does it cost?

Free until 22 October 2026. From then on it uses pay-per-event pricing: $0.30 per 1,000 URLs (url-result, $0.0003 each) plus Apify's standard $0.00005 Actor start fee. Platform usage is included. Use maxUrls or the run's maximum cost setting to cap spending.

🔌 Use it via API, Python or integrations

Run it from the Apify Console, on a schedule, or from code with the Apify API, the Python or JavaScript apify-client, or MCP. Connect results to Google Sheets, Slack, Zapier, Make, n8n or a webhook from the Integrations tab.

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("lanternlane-data/sitemap-url-extractor").call(run_input={"startUrls": ["apify.com"], "maxUrls": 10000})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

❓ FAQ

The site has no sitemap. The run reports it in RUN_SUMMARY. Most CMSs (WordPress, Shopify, Wix, Webflow, Ghost ...) publish one; try the sitemap URL directly if it's in an unusual place.

How do I get only new pages every day? Turn on onlyNewUrls and schedule the Actor. Connect Slack, e-mail or a webhook integration for alerts.

Does it crawl pages? No. It only reads the sitemap files the site publishes for crawlers, which is why it's so fast and cheap. Use the URL list as input for a crawler if you need page content.

Something broke? Open an issue and we'll fix it quickly.

🏮 More from Lantern Lane Data

Jobs & hiring data is our focus: job postings straight from employers' applicant tracking systems and official public feeds, in one consistent schema. More ATS platforms and hiring-signal tools are on the way.

• 💼 ATS Jobs Scraper: Every open job from 7 ATS platforms (Greenhouse, Lever, Ashby, Workday, SmartRecruiters, Workable, Recruitee) in one schema, with new and closed job alerts.
• 🌍 Remote Jobs Scraper: Remote jobs from 7 boards via their official public feeds, deduplicated, with salary and location filters.

Other tools

• 🛍️ Shopify Products Scraper: Every product, price, variant and stock level from any Shopify store, with price-change tracking.
• 🔦 AI Brand Visibility Tracker: How often ChatGPT, Perplexity, Gemini and Claude mention, rank and cite your brand vs competitors.

---

Made by Lantern Lane Data.