Sitemap URL Extractor: robots.txt, gzip, Changes
Pricing
from $0.30 / 1,000 urls
Extract every URL from a website's XML sitemaps: found through robots.txt, sitemap indexes and .gz files followed, lastmod and date filters, URL patterns, and new, changed or removed URLs since your last run. Reads sitemap files only, never the pages.
All notable changes to this Actor are documented here. The output fields are a contract: no field is renamed, removed or retyped outside a major version.
One website receives at most 4 requests at a time and 10 a second, sitemap downloads and status checks together, whatever the run does elsewhere.
A status check that returned nothing (timeout, refused address) is not charged.
New option Check the HTTP status of each URL (checkUrlStatus, off by default): a HEAD request per saved
URL, redirects reported and not followed, GET only when HEAD is not supported, internal addresses never
requested. New output fields statusCode, redirectUrl and statusError (null when the option is off),
and a URL status view. Charged as the url-status-check event, $0.70 per 1,000 checked URLs on top of
the URL itself; the budget counts both events of a checked URL.
RSS 2.0, RSS 1.0 and Atom feeds are read like sitemaps (link and publication or update date). Feed URLs
(/feed, /rss, .rss, .atom) are read directly.
Memory limit raised from about 450 to 1,000 URL entries per MB: the old figure came from a measurement distorted by the memory leak fixed in 0.1.5. Measured since on the platform: about 580 bytes per entry (885 MB of 1,024 at 1.5 million entries).
No existing output field changed.
First version.
/sitemap.xml, /sitemap_index.xml and /sitemap-index.xml..xml.gz files whatever their content type, plain-text
sitemaps, and the image, video, news and hreflang extensions.url, site, lastmod, changefreq, priority, sitemapUrl, discoveredFrom, imageCount,
videoCount, newsTitle, newsPublishedAt, alternates, change-tracking fields and scrapedAt.modifiedSince (absolute or relative), requireLastmod, and
sitemapUrlPatterns for the children of an index.trackChanges, onlyChanges): new, changed, unchanged and removed URLs, removals only
after complete reads. The state is stored gzipped, in chunks, in a named key-value store.maxItems (default 500), maxItemsPerSite, maxSitemapsPerSite, and the user's maximum cost per
run (pay-per-event), all claimed before each push. A website that reaches its limit stops being read.HttpCrawler, every sitemap body stayed in memory until the end of the run.SUMMARY record with the status of each website (ok, partial, no_sitemap, not_reached,
blocked, failed), with the HTTP status, content type and size of unexpected answers. A website that
answers with a bot challenge (AWS WAF, Cloudflare) is reported as blocked, not failed. The run fails
only when no website could be read for technical reasons.