First public build.
- Every URL from a site's sitemaps, with
lastmod paired to its URL.
- Classification into product / category / blog / page / other, trusting the
site's own sitemap filing before falling back to path shape.
- Gzipped sitemaps are read rather than skipped — which is where large
catalogues actually live.
- One free summary row per site with the totals by kind.
includeTypes filters while reading, so a narrowed run spends its budget on
what was asked for.- Two request budgets per site: how many sitemaps to read, plus an allowance
for unreachable ones, so a site with hundreds of dead sitemap entries cannot
run up the bill.
Deliberately not included: live status checking of every URL. At the
engine's per-host delay that turns a 10,000-URL run into a two-hour one, and
bulk-url-status-checker already does that job well. This actor reports what a
site publishes about itself.
