Sitemap Scraper - Find the Sitemap and Watch for New URLs
Pricing
from $0.73 / 1,000 url returneds
Sitemap Scraper - Find the Sitemap and Watch for New URLs
Give it a domain. It reads robots.txt to find where that site really keeps its sitemap, follows nested indexes, unpacks gzip, and returns one row per URL with lastmod, change frequency and priority. Monitoring mode returns only the URLs that are new since the last run.
Pricing
from $0.73 / 1,000 url returneds
Rating
0.0
(0)
Developer
NeverEmpty
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
2 days ago
Last modified
Categories
Share
Sitemap Scraper — find the sitemap, follow it, and watch for new URLs
Give it a domain. stripe.com is enough.
It reads robots.txt to find where that site actually keeps its sitemap, follows nested indexes, unpacks gzip,
and returns one row per URL with the last-modified date, change frequency and priority the sitemap states.
Why finding it matters
Asking for /sitemap.xml fails on a lot of real sites. Measured on 2026-09-07 against ten live sites:
| Site | /sitemap.xml | Where the sitemap actually is |
|---|---|---|
| stripe.com | 404 | robots.txt declares /sitemap/sitemap.xml |
| www.nytimes.com | 403 | robots.txt declares 25 separate sitemaps |
| www.shopify.com | 200 | robots.txt declares sitemaps_list.xml (a different name) |
| www.bbc.com | 200 | robots.txt declares 39 sitemaps |
| github.com | 406 | declares none — this Actor reports that instead of guessing |
| apify.com | 200 | an index pointing at 12 more files |
So the order is: read robots.txt first, take every Sitemap: line, and only guess if there are none.
Nine common locations are tried as a fallback, and a site with no sitemap is reported as no-sitemap with the
reason — not returned as an empty result.
Monitoring mode: only what is new
Turn on Monitoring mode and the Actor remembers every URL it returned.
- The first run returns everything and remembers it.
- Every run after that returns only URLs that were not there before.
- A quiet day returns one row saying nothing is new, and nothing is charged.
That makes a daily watch on a competitor's site, a documentation site or a job board cheap: you pay for new pages, not for the whole sitemap every day.
What comes back
| Field | What it is |
|---|---|
site, url | The site you asked for, and the URL found in its sitemap |
host, path, depth | Split out so you can group by section. depth is how many path segments deep |
extension | pdf, html, … or null when the URL has none |
lastmod, lastmodAt | Exactly as the sitemap wrote it, and the same value as an ISO instant |
changefreq, priority | What the sitemap claims. null when it says nothing |
sitemapUrl, sitemapDepth | Which file this URL came out of, and how many indexes deep it was |
foundVia | robots.txt, common-path or given — how the sitemap was located |
sitemapIsGzipped | Whether that file was served compressed |
status, reason | ok, or why a site produced nothing |
A field the sitemap does not state comes back as null. Nothing is filled in with a zero or a guessed date.
Filters
All of these run on URLs already fetched — nothing extra is downloaded to apply them.
- Path contains / does not contain — keep
/blog/, drop/tag/ - File extensions — only
pdf, onlyhtml - Changed on or after / before — by the sitemap's own
lastmod. ⚠️ A URL with no date is kept, because unknown is not the same as old - Minimum priority, Change frequency, Keywords / Exclude keywords
Limits you control
Maximum URLs, Maximum sitemap files and Maximum index depth. Large sites split their sitemap into hundreds
of files, so these keep a run bounded and predictable.
Pricing
Charged per URL returned. Sites that produced nothing — no sitemap, a sitemap that answered 403, a run where nothing is new in monitoring mode, or filters that matched nothing — come back with the reason and are not charged.
Notes
- gzip is detected by the file's own first two bytes, not by the
.gzin the name, because some servers compress without saying so in the URL. - The same sitemap file is never read twice in one run, and the same URL is never returned twice.
- Only
httpandhttpsare accepted. Anything else is rejected rather than rewritten into a different host.
