Extract URLs from a site's XML sitemaps with lastmod, image counts and hreflang. Capped at 1,000 URLs per site by default (raise maxUrlsPerSite, 0 = no limit). Auto-finds sitemaps, filters by regex and date. $0.20 per 1,000 URLs.
Unreleased: default mode for an empty input (GAP8)
BREAKING (default change): the default maxUrlsPerSite is now 1,000 (was 10,000), in the input schema and in code when the field is left out. A run that relied on the old default now stops after 1,000 URLs per site, at most $0.20 per site. Set maxUrlsPerSite to 10000, or 0 for no limit, to get the old behaviour.
An empty input ({}) no longer fails with HTTP 400 or a missing-field error. startUrls is no longer required and now has a schema default (and prefill) of ["https://www.nasa.gov/wp-sitemap.xml"]. When startUrls is empty or is exactly that sample, the run is in default mode: it reads the NASA sitemap, returns at most 40 URLs (a smaller maxUrlsPerSite wins), logs Default mode: ..., adds defaultMode and defaultStartUrl to RUN_SUMMARY and prefixes the status message with Default mode (sample sitemap): .
Input schema: the prefill changed from nasa.gov and the Cloudflare blog sitemap to the NASA sample sitemap, so the Store test runs default mode. Runs that list any other sites behave as before, apart from the new 1,000 default cap.
Lint only: removed unused imports in common.py (pyflakes clean). No behaviour change.
Unreleased: unreachable sitemaps are an error, robots.txt on every redirect
A site where not one sitemap could be read (every sitemap returned an error or could not be reached) now has status: "error" in RUN_SUMMARY, with the last error, instead of ok. If every sitemap was blocked by robots.txt, the status is skipped. A site where some sitemaps load and others fail stays ok, with the failures counted in sitemapErrors.
Redirects are now followed one hop at a time, and robots.txt is checked for every hop. A sitemap that redirects to a disallowed path or to another host whose robots.txt disallows it is not fetched. Up to 5 redirects are followed.
If a site's robots.txt can't be read (a server error after one retry, a network failure, or a 200 that is an HTML page or not text/plain), the site is skipped for the run with a warning and status: "skipped". Before, it was read as if it had no rules. A missing robots.txt (any 4xx) still means no rules.
Unreleased: URLs without a lastmod
With lastmodAfter or lastmodWithinDays set, URLs without a lastmod are dropped by default and counted (undatedDropped). Set keepUndatedUrls to true to keep them (undatedKept).
