Sitemap URL Extractor & New Page Monitor

Extract every URL from any website's sitemap in seconds — just enter a domain. The Actor finds the sitemaps for you (robots.txt, /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, .xml.gz, text sitemaps), follows nested sitemap indexes, and returns a clean list of page URLs with lastmod, changefreq, priority, hreflang alternates, image, news and video data.

Turn on monitor mode and schedule it to get only the new pages a website published since the last run — new products, blog posts, landing pages, job pages or docs from your competitors.

No browser, no proxies, no login: it only reads the public sitemap files that websites publish for crawlers, so it is fast and cheap.

What can it do?

• 🔎 Auto-discovers sitemaps from robots.txt and common locations; or give it a sitemap URL directly
• 🗂️ Follows sitemap indexes (any depth), gzip (.xml.gz), text and RSS/Atom sitemaps
• 🧾 Rich output: lastmod, changefreq, priority, hreflang alternates, image URLs, Google News title and publication date, video title
• 📅 Date filter: only URLs modified in the last N days or after a date — and it skips old child sitemaps entirely, so big sites stay fast and cheap
• 🎯 Regex include / exclude filters (e.g. only /blog/, drop /tag/)
• 🆕 Monitor mode: remembers URLs between runs and outputs only new ones (perfect with Apify Schedules + email/Slack/webhook integrations)
• ✅ Optional HTTP status check (HEAD request) to find 404s and redirects listed in sitemaps
• 🛡️ Robust: retries with exponential backoff, honors Retry-After, per-site error report, one broken site never fails the whole run
• 🤖 AI-agent friendly: simple input, flat JSON output, usable via the Apify API and MCP server

Use cases

• SEO audits — get the full list of indexable URLs, find 404s/redirects in sitemaps, check hreflang coverage
• Competitor monitoring — get notified when a competitor publishes new pages, products, pricing or landing pages
• Content and news tracking — collect fresh articles from news sitemaps every hour or day
• Crawl planning / RAG — feed a clean URL list into Website Content Crawler or your own scraper instead of crawling blindly
• Site migrations — export all old URLs to build redirect maps
• E-commerce — list every product URL of a store, or only products added this week

Input

Only startUrls is required. Example:

{
  "startUrls": ["apify.com", "https://www.bbc.co.uk/sitemaps/https-index-uk-news.xml"],
  "maxUrlsPerSite": 100,
  "includeUrlPatterns": ["/news/"],
  "modifiedWithinDays": 7,
  "onlyNewUrls": false,
  "checkStatus": false
}

Field	Description
startUrls	Domains, homepage URLs, robots.txt URLs or direct sitemap URLs. A homepage/path URL is treated as its whole domain — use includeUrlPatterns to narrow it down.
maxUrlsPerSite	Max URLs output per site (default 100).
includeUrlPatterns / excludeUrlPatterns	Case-insensitive regular expressions.
modifiedWithinDays / modifiedAfter	Keep only URLs with a recent <lastmod>.
keepUrlsWithoutLastmod	Keep undated URLs when a date filter is set.
onlyNewUrls	Monitor mode: output only URLs not seen in earlier runs. The first run saves a baseline.
emitAllOnFirstRun	Output everything on the first monitor run too.
checkStatus	Adds statusCode, finalUrl, redirected.
Advanced	maxSitemapsPerSite, maxDepth, maxConcurrency, requestTimeoutSecs, maxRetries, proxyConfiguration.

Output

One dataset item per URL. Export as JSON, CSV, Excel, XML or via API. Example:

{
  "site": "techcrunch.com",
  "url": "https://techcrunch.com/2026/10/30/nvidia-weighs-playing-the-cosigner-for-nscales-ai-data-center-in-west-virginia/",
  "lastmod": "2026-10-31",
  "changefreq": null,
  "priority": null,
  "sitemapUrl": "https://techcrunch.com/sitemap-page-122.xml",
  "imageCount": 1,
  "images": ["https://techcrunch.com/wp-content/uploads/2026/03/GettyImages-2235971874.jpg"],
  "extractedAt": "2026-11-04T02:57:47Z"
}

Optional fields appear when available: alternates ([{"hreflang": "de", "href": "..."}]), newsTitle, newsPublicationDate, videoTitle, isNew (monitor mode), statusCode / finalUrl / redirected (status check).

A per-site report (sitemaps found, URLs found/filtered/output, new URLs, errors) is saved to the key-value store record SUMMARY.

How to monitor a website for new pages

1. Add the sites to startUrls and enable Monitor mode.
2. Run once — this saves the baseline (no results, no charge for URLs).
3. Create a Schedule (e.g. daily) in Apify Console for the saved task.
4. Add an integration (email, Slack, Google Sheets, Zapier, Make, webhook) to receive the new URLs.

The list of seen URLs is stored as hashes in a private named key-value store sitemap-url-monitor-state in your own Apify account. Delete a record there to reset the baseline for a site.

Pricing

Pay per result: you pay only for URLs written to the dataset. Sitemap discovery, sites without sitemaps, filtered-out URLs and the monitor-mode baseline run are not charged per URL. Set a maximum cost per run in the run options — the Actor stops cleanly when it is reached.

FAQ

The site has no sitemap — what happens? The site is reported as no-sitemap-found in SUMMARY; the run continues with the other sites. This Actor does not crawl HTML pages — use a crawler for sites without sitemaps.

Does it find every page of a website? It returns every page the site lists in its sitemaps. Most CMSs (WordPress, Shopify, Wix, Squarespace, Webflow, Next.js, Docusaurus …) publish complete sitemaps, but pages left out of the sitemap won't appear.

How big can a site be? Millions of URLs are fine. Raise maxUrlsPerSite and maxSitemapsPerSite. Use the date filter to make large-site runs much cheaper.

Is it legal? Sitemaps are public files that site owners publish specifically so that automated tools can discover their pages. The Actor collects only URLs and sitemap metadata — no personal data. You are responsible for how you use the output and for respecting each website's terms.

Do I need proxies? Usually not. If a site blocks requests to its sitemap, enable Apify Proxy in the Advanced section.

Can I use it from code or an AI agent? Yes — call it via the Apify API, the Python/JS clients, or the Apify MCP server. Input and output are simple JSON.

Something doesn't work? Open an issue on the Issues tab with the site URL — fixes are usually quick.