Sitemap Monitor: All URLs + New Pages Since Last Run
Pricing
Pay per event
Sitemap Monitor: All URLs + New Pages Since Last Run
Extract every URL from any website's XML sitemaps (robots.txt discovery, indexes, .xml.gz, hreflang, images) and schedule it to get only NEW pages since the last run. Optional HTTP status and redirect check.
Pricing
Pay per event
Rating
0.0
(0)
Developer
Lindenbyte Studio
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
an hour ago
Last modified
Categories
Features
Share
Sitemap Monitor: Extract All URLs + Get New Pages Since Last Run
Point it at any website and get every URL from its XML sitemaps: robots.txt discovery, sitemap indexes, nested indexes, .xml.gz, plain-text sitemaps, lastmod, changefreq, priority, hreflang alternates and image counts. Then switch on monitoring mode and schedule it: each run outputs only the pages that are new since the last run. Optionally check the HTTP status and redirects of every URL.
Why this one
- New-pages feed, not just a dump:
onlyNewremembers every URL per site and returns only the new ones. Schedule it daily or hourly to see when a competitor launches a product, publishes a post or adds a landing page. - Cheap to start monitoring: turn off
emitOnBaselineand the first run just records the baseline (no URLs output, nothing charged per URL). - Finds the sitemaps for you: give
example.comand it reads robots.txtSitemap:lines, then falls back to/sitemap.xml,/sitemap_index.xml,/sitemap-index.xml. Or pass sitemap URLs directly. - Handles real-world sitemaps: indexes up to 10 levels deep, gzip, text sitemaps, hreflang (
xhtml:link), image sitemaps, big sites (hundreds of thousands of URLs). - Filter before you pay: regex
urlPattern(only/blog/or/products/) andchangedSinceDays(bylastmod; old child sitemaps are not even downloaded). Filtered URLs are free. - Status check built in: HEAD request (GET fallback) per URL with status code, final URL and redirect count. Finds 404s and redirect chains listed in your own sitemap. Polite: respects robots.txt and Crawl-delay, per-host throttle.
- Pure HTTP, no browser: fast and cheap to run. Optional proxy only for sites that block datacenter IPs (403/429): just those sites are retried through it, everything else stays direct.
Use cases
- Competitor monitoring: new product pages, blog posts, landing pages, locations, job pages.
- SEO audits: full URL inventory, sitemap URLs that 404 or redirect, stale
lastmod, hreflang coverage. - Content teams and agencies: weekly list of what clients or competitors published.
- Crawling pipelines: feed only new URLs into a scraper instead of re-crawling the whole site.
- E-commerce: new SKUs on a competitor's store the day they go live.
Input
| Field | What it does |
|---|---|
sites | Domains (example.com), homepage URLs or sitemap URLs (.xml, .xml.gz, .txt) |
onlyNew | Monitoring mode: output only URLs not seen on the previous run (state per site) |
emitOnBaseline | Monitoring mode, first run: output all URLs (on) or only record the baseline (off) |
stateKey | Name of the monitor; different names keep separate memories |
urlPattern | Case-insensitive regex; only matching URLs are output |
changedSinceDays | Only URLs whose lastmod is within the last N days |
checkStatus | Add statusCode, finalUrl, redirectCount per URL |
maxUrlsPerSite | Cost cap: max URLs output per site per run (default 10,000) |
maxScannedUrlsPerSite | Safety cap on URLs read per site (default 200,000; reading is free) |
statusConcurrency, perHostDelayMs | Status-check speed and politeness |
maxSitemapDepth | Nested sitemap index levels to follow (default 3) |
Example: daily new-pages monitor
{"sites": ["competitor.com", "https://other.com/sitemap_index.xml"],"onlyNew": true,"emitOnBaseline": false,"urlPattern": "/(blog|products)/"}
Create a schedule (Console > Schedules) and connect an integration (email, Slack, webhook) to get the new URLs each day.
Output
One row per URL (real example):
{"site": "crawlee.dev","url": "https://crawlee.dev/blog/archive","lastmod": null,"changefreq": "weekly","priority": 0.5,"isNew": null,"sitemapUrl": "https://crawlee.dev/sitemap.xml","alternates": [],"alternatesCount": 0,"imageCount": 0,"statusCode": 200,"finalUrl": "https://crawlee.dev/blog/archive","redirectCount": 0,"statusError": null,"scrapedAt": "2026-11-13T10:25:38.920Z"}
isNew is true in monitoring mode, null otherwise. alternates lists hreflang { hreflang, href } pairs. A per-site summary (sitemaps fetched, URLs scanned, new URLs, filtered counts, errors) is saved to the key-value store record SUMMARY.
Pricing
Pay per result: you pay per URL saved to the dataset and per successful status check, plus a small start fee per run. URLs that are filtered out, already seen, or recorded as a silent baseline are free. If you enable the optional proxy, each request actually sent through it is one proxied-request event (sites that don't block you are never proxied, so they cost nothing extra). Set a max charge per run in the run options to cap cost.
Limits (honest)
- Only pages listed in the site's sitemaps are found. Sites without a sitemap return an error for that site (it is not a crawler).
- "New" means "new in the sitemap since the last run". Removed URLs are not reported in this version.
- Some sites block datacenter IPs on their sitemaps (e.g. some news sites answer 403). Enable the optional proxy (residential works best) to retry just those sites; sites with browser-only bot challenges still fail with a clear error in the log and summary.
- Monitoring state lives in your account's named key-value store
sitemap-monitor-state. ChangingstateKeystarts a fresh baseline.
Legal
Reads public sitemaps and robots.txt that websites publish for automated discovery; status checks respect robots.txt. Do not use it to overload websites. Not affiliated with any website it reads.