# Changelog of Regulatory Page Change Monitor – Diffs & Alerts (`lwsdjfls/regulatory-page-change-monitor`) Actor

- **URL**: https://apify.com/lwsdjfls/regulatory-page-change-monitor/changelog.md
- **Full Actor documentation**: https://apify.com/lwsdjfls/regulatory-page-change-monitor.md

## Changelog

### 0.1

- Initial release.
- A run without pages says to add one to "URLs to monitor", with an example, instead of "expected array, received undefined".
- "Charges nothing for changes" is replaced by "charges for no change": Apify's start event is charged for every run.
- README corrected: a run stopped by a structure change still pays Apify's start event; the Actor sends a Chrome browser user agent and does not identify itself as a bot; the limit of 2 requests per second per host applies only without JavaScript rendering, on average and in bursts of up to `maxConcurrency`. Removed claims the repository cannot back: the 25 to 35% XPath memory figure, the response time for support, the Slack and Google Sheets integrations and "daily is typical". Added the worst case per run at the default limits ($5.50005, or $5.5001 with JavaScript rendering), a tip to set Max cost per run in Apify Console, and two limitations: the default `timestamps` and `counters` filters can hide a real change, and a page in "URLs to monitor" that stops loading is not reported as removed.
- When none of the URLs to monitor can be read because of a temporary error, such as a timeout, a refused or dropped connection, or HTTP 429, 500, 502, 503 or 504, the run no longer fails after 4 attempts per URL (3 with Render JavaScript): it keeps trying the first of them for up to 10 minutes, once per run and never past its timeout, and once that URL loads it tries the others again. A run in which any URL could be read never waits, and a URL that answers 404 still fails at once. With Render JavaScript, a page that keeps failing on a temporary browser error now fails with a message that names the URL and the number of attempts.
- Close to its timeout, a run stops starting new pages and delivers what it found, instead of timing out with nothing delivered. It keeps time free to deliver what it found (0.1 s per record, from a measurement on Apify) and to finish the pages it already started, and a page's retries stop in time for that. With XPath selectors it also keeps time free for the extraction of every page in flight, which runs one page at a time: as long as the longest extraction of the run so far took (a second before the first), not the 30-second limit of an extraction. That limit, kept free for every page in flight, came to ten minutes at a `maxConcurrency` of 20, and in a test of the README's example (41 pages that each answer in 5 seconds) a run with a 5-minute timeout stopped too early, dropped the first check and delivered nothing, although its work fitted. Near the end an extraction gets only the time that is left, at least a second, and a page whose extraction did not finish in it is not checked rather than failed. With Render JavaScript the run likewise keeps free as long as the waits for the network to go quiet and for `waitForSelector` took so far (a second before the first page), instead of 10 seconds plus the whole `requestTimeoutSecs` (up to 120 seconds); near the end each wait gets only the time that is left, at least a second, and a page whose wait ran out of it is not checked, instead of reported as a structure change or read before it was complete. A page that no longer fits with its whole `requestTimeoutSecs` starts with less time to load, at least 5 seconds, and the first page of a run always starts. URLs to monitor that it did not reach are left as they were (not reported as removed, not charged), linked pages it did not reach are not reported as removed, and the log says how many were left and suggests a longer timeout or splitting the URLs. The URLs to monitor are now checked least recently checked first, and URLs never checked before all others; otherwise the input order is kept. A run that reports no change and records no new baseline can take a day to save when it checked its URLs, so until then runs more often than daily may start with the same URLs. Duplicate URLs in a long list are found in one pass instead of comparing every URL with every other.
- URLs to monitor that a run leaves out because it reached `maxPagesPerRun` or its maximum cost are left as they were, and later runs start with them. Before, the same first URLs were checked in every run and the others never, and every new URL among them was stored without content, so that once it was checked its page was reported, and charged, as added. This now includes the URL at which the run reaches its maximum cost: it was saved as checked without content, so a later run reported its whole content as added and charged for it, or, if it had been checked before, it moved to the back of the order without a check. It is now dropped from the run and left as it was.
- `requestTimeoutSecs` is described as the timeout of one attempt to load a page, which it always was.
- A first check (baseline) of a URL that the run's timeout cuts short is dropped: nothing of it is saved, delivered or charged, and later runs start with that URL, as with every URL not checked yet. Before, it was saved half-way, so the next run that reached the pages it had missed reported them as added, charged them and sent them to the webhook. The run does not fail when that URL was its only one. Such a URL does not count as checked, though, so a run in which every other URL to monitor failed still fails, with their errors; the README's Limitations said before that a run fails only when none of the URLs could be read.
- A page that the run's time limit cut short, because its shortened attempt got no answer or its retries stopped to end in time, counts as not checked instead of failed: a URL to monitor is left as it was and counted as not processed, a linked page leaves its URL's check incomplete (no removals), and a first check is dropped. A run whose only URL was cut short no longer ends as failed. A page that used all its attempts, or whose server asked to wait longer than allowed, still counts as failed. With Render JavaScript, the error of a page that kept answering with a temporary HTTP status now gives the number of attempts made.
- Near the end of a run, a page that fails on a temporary error, such as HTTP 503 or a dropped connection, is tried again as long as an attempt of at least 5 seconds still fits in the time left for it, and that attempt gets the time left. Before, a retry started only when a whole attempt with the full `requestTimeoutSecs` still fitted, so a page among the last of a run that had time for it was not tried again and counted as not checked: a first check was then dropped, so the run delivered and charged nothing, and a later check lost its removals. With a timeout the Actor now makes the attempts at a page itself, one request each, with the HTTP client's retry policy, temporary statuses and Retry-After, so that each attempt ends in time. A page whose server keeps failing uses all its attempts and has failed, as it has without a timeout.
- The run's status message counts the URLs to monitor that a run left out, as "N not processed (the run's time limit was near)" or "(maxPagesPerRun or the maximum cost was reached)", and such a run ends as partial.
- The one patient wait for a URL to monitor that is down now leaves free the time its last attempt and the delivery need, so the log says how long it really waits.
- The default run timeout is 22 hours instead of one hour (`runTimeoutSecs` 79,200 in store/listing.json): the largest run the input allows checks 100,000 pages (the maximum of `maxPagesPerRun`), 50,000 s at 2 requests per second to one site, and delivers 200,000 records (a page-checked charge and a baseline or change per page) in 20,000 s at 0.1 s each; 70,000 s plus 10% is 21.4 hours, rounded up to 22. A list page delivers a record per item, so a first run over that many list pages needs longer.
- The state key (`stateKey`) is a secret input: stored encrypted, like the webhook URL, because it often names a customer. Apify checks no pattern on a secret field, so any text is now accepted instead of failing the run: a key of letters, digits, dashes and underscores keeps the state it had, other text gets a state of its own, spaces around it are ignored and a blank one counts as none.
- Store discounts: Apify's paid plans pay less for every event except the Actor start, 5% on the Bronze tier, 10% on Silver and 15% on Gold and higher. The prices in the README are the list prices, which the Free plan pays.