# Changelog of Sitemap Extractor: Every Page URL, with Change Alerts (`rookdev/sitemap-change-monitor`) Actor - **URL**: https://apify.com/rookdev/sitemap-change-monitor/changelog.md - **Full Actor documentation**: https://apify.com/rookdev/sitemap-change-monitor.md ## Changelog ### Unreleased (listing only) - Store title changed to lead with the plain job, not monitoring (2 Nov 2026; see docs/research/2026-11-02-store-crowding.md). The Actor name in its URL and all behaviour are unchanged. ### 0.2 (2026-10-30) - **Updated pages are opt-in.** New input `reportUpdatedPages` (default off) replaces `reportChanged` (which was on by default and is now ignored). In the owner's field test, Shopify stores (allbirds, colourpop, glossier) re-dated their pages all the time: a run one minute after another reported 2,338 pages as changed (colourpop 1,072 of 1,318). The option is part of the monitoring memory's key, so turning it on starts a fresh memory; leaving it off keeps existing memories. - **Re-stamps are ignored.** With updated pages on, a sitemap file in which half or more of its known pages (at least 50) get a new `` at once reports none of them, and the same test applies to the whole site. `RUN_SUMMARY` lists the files under `restampedSitemaps`, and the status message says how many dates were ignored. - **Capped monitors:** pages seen for the first time in a read cut by the URL cap are reported `new` only when their `` is after the previous run; the rest are recorded for free. Before, reordered sitemaps were charged as new pages run after run. Pages the monitor could not see are forgotten after 30 days unseen. - **Pricing:** one `site-checked` per site, plus one more per started 50,000 URLs read after the first 50,000 (billed up to the URL cap, or 10 times it with a URL filter). Sites with the default cap still pay one check. Failed sites are never charged. - **Cost protection** (after the cost-abuse audit): a run-wide cost guard counts compute and every row written, and stops new work once the run is $0.02 ahead of what it has earned. It is checked before every site, request and retry wait, and every 1,000 entries while parsing. At most 1,000 websites per run, sent in one push of free rows. 50 failed sites in a row end the run. A site is read only when the spending limit still covers one check and one URL, counting the sites already being read. Sites not checked share one free `skipped` row per reason. - **Bounded work:** a URL filter scans at most 10 times the URL cap; filter lists are capped at 100 entries of 200 characters. robots.txt is read up to 512 KB. Sitemap files over 4 MB (or 16 MB unpacked) are downloaded and parsed in a few memory-sized slots (1 at 512 MB, 2 at 1,024 MB), and fewer 100,000-URL sites run at once, so big runs fit their memory. - **Network safety:** private, local, link-local, carrier-NAT, multicast and reserved addresses are refused, in any spelling (`127.1`, `0x7f.1`, `169.254.43518`, IPv6-mapped). They are checked at input, again after DNS (every address, and the connection goes to the checked address) and on every redirect. A 10 s connect timeout and a 60 s deadline per request apply, with at most 2 attempts after network errors. At most 2 requests in flight per host, and a host that fails 10 times in a row is left alone for the run. - **24-hour cap per Apify account:** the cost guard's $0.02 per-run allowance now comes out of $0.20 of unpaid compute per account per 24 hours, across runs (template `AllowanceLedger`, one record in the fixed store `sitemap-change-monitor-limits`), so runs started back to back cannot add up to a large unpaid bill. A run stopped by it says so in plain words in its free `skipped` row and status message ("this Apify account has used today's allowance...; it resets within 24 hours"). If the record cannot be read, the run goes ahead. ### 0.1 (2026-10-29) - First release: finds sitemaps through robots.txt, `/sitemap.xml` and `/sitemap_index.xml`; reads sitemap indexes, gzip and plain-text sitemaps. - Monitoring of new, changed (lastmod moved, with the previous date) and removed pages; optional quiet first run; URL contains / excludes filters; URL cap per site. - Safe billing and monitoring: a site check is charged only after its rows are delivered; delivered rows are never charged twice; partial reads never report pages as removed; one broken site never fails the run. - Hardened after an independent review: per-sitemap "gone" safety, stream parsing with size caps, odd encodings and dates, robots.txt quirks, gzip edge cases, URL encoding, bounded retries, serialised billing. Rows carry a one-line `summary`; the status message lists the first changes for Slack alerts.