Sitemap Monitor: All URLs + New Pages Since Last Run avatar

Sitemap Monitor: All URLs + New Pages Since Last Run

Pricing

Pay per event

Go to Apify Store
Sitemap Monitor: All URLs + New Pages Since Last Run

Sitemap Monitor: All URLs + New Pages Since Last Run

Extract every URL from any website's XML sitemaps (robots.txt discovery, indexes, .xml.gz, hreflang, images) and schedule it to get only NEW pages since the last run. Optional HTTP status and redirect check.

Pricing

Pay per event

Rating

0.0

(0)

Developer

Lindenbyte Studio

Lindenbyte Studio

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

an hour ago

Last modified

Features

Limited permissions

Share

Sitemap Monitor: Extract All URLs + Get New Pages Since Last Run

Point it at any website and get every URL from its XML sitemaps: robots.txt discovery, sitemap indexes, nested indexes, .xml.gz, plain-text sitemaps, lastmod, changefreq, priority, hreflang alternates and image counts. Then switch on monitoring mode and schedule it: each run outputs only the pages that are new since the last run. Optionally check the HTTP status and redirects of every URL.

Why this one

  • New-pages feed, not just a dump: onlyNew remembers every URL per site and returns only the new ones. Schedule it daily or hourly to see when a competitor launches a product, publishes a post or adds a landing page.
  • Cheap to start monitoring: turn off emitOnBaseline and the first run just records the baseline (no URLs output, nothing charged per URL).
  • Finds the sitemaps for you: give example.com and it reads robots.txt Sitemap: lines, then falls back to /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml. Or pass sitemap URLs directly.
  • Handles real-world sitemaps: indexes up to 10 levels deep, gzip, text sitemaps, hreflang (xhtml:link), image sitemaps, big sites (hundreds of thousands of URLs).
  • Filter before you pay: regex urlPattern (only /blog/ or /products/) and changedSinceDays (by lastmod; old child sitemaps are not even downloaded). Filtered URLs are free.
  • Status check built in: HEAD request (GET fallback) per URL with status code, final URL and redirect count. Finds 404s and redirect chains listed in your own sitemap. Polite: respects robots.txt and Crawl-delay, per-host throttle.
  • Pure HTTP, no browser: fast and cheap to run. Optional proxy only for sites that block datacenter IPs (403/429): just those sites are retried through it, everything else stays direct.

Use cases

  • Competitor monitoring: new product pages, blog posts, landing pages, locations, job pages.
  • SEO audits: full URL inventory, sitemap URLs that 404 or redirect, stale lastmod, hreflang coverage.
  • Content teams and agencies: weekly list of what clients or competitors published.
  • Crawling pipelines: feed only new URLs into a scraper instead of re-crawling the whole site.
  • E-commerce: new SKUs on a competitor's store the day they go live.

Input

FieldWhat it does
sitesDomains (example.com), homepage URLs or sitemap URLs (.xml, .xml.gz, .txt)
onlyNewMonitoring mode: output only URLs not seen on the previous run (state per site)
emitOnBaselineMonitoring mode, first run: output all URLs (on) or only record the baseline (off)
stateKeyName of the monitor; different names keep separate memories
urlPatternCase-insensitive regex; only matching URLs are output
changedSinceDaysOnly URLs whose lastmod is within the last N days
checkStatusAdd statusCode, finalUrl, redirectCount per URL
maxUrlsPerSiteCost cap: max URLs output per site per run (default 10,000)
maxScannedUrlsPerSiteSafety cap on URLs read per site (default 200,000; reading is free)
statusConcurrency, perHostDelayMsStatus-check speed and politeness
maxSitemapDepthNested sitemap index levels to follow (default 3)

Example: daily new-pages monitor

{
"sites": ["competitor.com", "https://other.com/sitemap_index.xml"],
"onlyNew": true,
"emitOnBaseline": false,
"urlPattern": "/(blog|products)/"
}

Create a schedule (Console > Schedules) and connect an integration (email, Slack, webhook) to get the new URLs each day.

Output

One row per URL (real example):

{
"site": "crawlee.dev",
"url": "https://crawlee.dev/blog/archive",
"lastmod": null,
"changefreq": "weekly",
"priority": 0.5,
"isNew": null,
"sitemapUrl": "https://crawlee.dev/sitemap.xml",
"alternates": [],
"alternatesCount": 0,
"imageCount": 0,
"statusCode": 200,
"finalUrl": "https://crawlee.dev/blog/archive",
"redirectCount": 0,
"statusError": null,
"scrapedAt": "2026-11-13T10:25:38.920Z"
}

isNew is true in monitoring mode, null otherwise. alternates lists hreflang { hreflang, href } pairs. A per-site summary (sitemaps fetched, URLs scanned, new URLs, filtered counts, errors) is saved to the key-value store record SUMMARY.

Pricing

Pay per result: you pay per URL saved to the dataset and per successful status check, plus a small start fee per run. URLs that are filtered out, already seen, or recorded as a silent baseline are free. If you enable the optional proxy, each request actually sent through it is one proxied-request event (sites that don't block you are never proxied, so they cost nothing extra). Set a max charge per run in the run options to cap cost.

Limits (honest)

  • Only pages listed in the site's sitemaps are found. Sites without a sitemap return an error for that site (it is not a crawler).
  • "New" means "new in the sitemap since the last run". Removed URLs are not reported in this version.
  • Some sites block datacenter IPs on their sitemaps (e.g. some news sites answer 403). Enable the optional proxy (residential works best) to retry just those sites; sites with browser-only bot challenges still fail with a clear error in the log and summary.
  • Monitoring state lives in your account's named key-value store sitemap-monitor-state. Changing stateKey starts a fresh baseline.

Reads public sitemaps and robots.txt that websites publish for automated discovery; status checks respect robots.txt. Do not use it to overload websites. Not affiliated with any website it reads.