A run whose baseline store cannot be opened no longer dies. It used to call Actor.fail(),
so a customer — or Apify's automated quality test, which runs this Actor on their account —
saw a failed run with no output at all. That is what flagged the Website Change Monitor under
maintenance on 2026-08-22 and the App Store Review Monitor on 2026-08-28, and this Actor's public
runs show the same signature. It now falls back to the run's own key-value store, publishes every
product as a first sighting, puts the new STATE_STORE_UNAVAILABLE warning on every row, and says
so in the run's status message. It never claims a change. STATE_UNAVAILABLE is still raised
when the named store and the run's own store both refuse.
A first run now produces rows. With onlyReportChanges on — the default — a first sighting
was treated as "nothing changed" and nothing was pushed, so a customer's first run ended with an
empty dataset. That contradicted this Actor's README ("every product is a baseline row") and its
charge tiering, which prices a baseline as a check because "the customer got a starting point out
of it". The Store's daily auto-test is the sharper consequence: it flags an Actor whose default
input yields an empty dataset three days running as under maintenance. A first sighting is now
published and charged as a check; a genuinely quiet check still publishes nothing and still
advances the baseline.
robots.txt rules written with wildcards are now obeyed. The shared policy used Python's
urllib.robotparser, which implements neither * nor $ in path patterns, so every rule of the
form Disallow: /path/* — an entirely ordinary way to write one — was read as permission. It is
replaced by a spec-correct matcher (agent_tools_core.robots) supporting *, $,
longest-pattern-wins precedence with Allow breaking ties, and per-user-agent groups.
This can change what a run reads. Addresses that were fetched before may now be skipped, on
sites whose robots.txt asked for that all along.