# Changelog of Zoopla Scraper — UK Property Listings (`devilscrapes/zoopla-uk-property-scraper`) Actor

- **URL**: https://apify.com/devilscrapes/zoopla-uk-property-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/devilscrapes/zoopla-uk-property-scraper.md

## Zoopla UK Property Scraper — Changelog

### 0.7.2 — 2026-09-29 (banked on main; push blocked by the invoice 403)

- **Fix: a 5-minute run had a 0-second budget.** `_resolve_run_budget`
  subtracted a flat 300 s margin from `timeout_at`; Apify's automated Store
  QA runs the prefill with a 300 s timeout, so the budget was 0 before the
  first listing, the loop declared "budget exhausted, 28 listing(s)
  unprocessed" 41 s after boot, and the run FAILED with zero rows (run
  `QjBoUDx5cpjMItqer`, 2026-09-28 14:07Z) — which triggered the
  "deprecated in 14 days" notice. Any customer `timeoutSecs` ≤ ~330 s hit
  the same wall. The margin is now 20 % of the real remaining time,
  floored at 20 s and capped at the old 300 s, so a QA run keeps 240 s and
  a full 3600 s run keeps the same reserve as before.
- The budget-exhausted log line now prints the budget actually in force
  instead of the 3300 s fallback constant.

### 0.7 — 2026-09-17

- **Fix: detail-page enrichment was always null — customer-facing, not
  merely "pre-existing."** 0.6's own QA run (`ewloPhfirZhNGaYM4`) had
  `enrichDetails` ON (schema default AND prefill), fetched all 3 detail
  pages successfully (`goto:` succeeded on every one), and still produced
  0/16 detail fields on every row — `description`, `epc_rating`,
  `epc_graph_url`, `agent_name`, `agent_phone`, `agent_url`, `photo_urls`,
  `postcode`, `latitude`, `longitude`, `property_type` were null on every
  customer row since 0.1, while the Store description and README output
  table both advertise exactly those fields ("price, address, bedrooms,
  agent phone, floor plan, EPC, photos, full description"). Customers have
  been charged the `result-row` PPE event for rows missing the columns the
  listing sells.
  - Root cause, read directly from `DETAIL_PARSE_MISS_HTML` (the KVS dump
    that run's own fault-isolation path already captured): Zoopla's detail
    page carries no `__NEXT_DATA__` blob (`parse_detail_json` always
    misses) and its DOM has no `data-testid` hooks for any of these fields
    (`parse_detail_dom` always misses too) — the page streams a React
    Server Components payload instead
    (`<script>self.__next_f.push([1,"..."])</script>` chunks). The real
    listing payload is one flat `{"__typename":"ListingData",...}`
    component-props object embedded in that stream, with `description`
    offloaded to a separately-referenced text chunk.
  - Fix: `src/parsers/common.py` gained `extract_rsc_payload()` (concatenate
    every RSC chunk — each is independently valid JSON),
    `extract_rsc_listing_data()` (balanced-brace-match the `ListingData`
    object out of the concatenated stream — no full React Flight protocol
    needed), `resolve_rsc_text_ref()` (resolve a `"$<id>"` reference against
    its `<id>:T<hexlen>,<text>` chunk), `html_to_text()` (strip the HTML
    fragment `description` ships as), and `zoocdn_photo_url()`/
    `rsc_photo_urls()` (build full-size CDN photo URLs from the bare
    filenames the payload carries — confirmed against the same filename
    hash appearing as a complete URL in the page's own JSON-LD `image`
    field). `string_list()` now also unescapes HTML entities (RSC feature
    bullets ship raw `&amp;`).
  - `src/parsers/detail.py` gained `parse_detail_rsc()`, wired into
    `scraper.py`'s existing fallback chain as the now-load-bearing middle
    tier: `parse_detail_json(html) or parse_detail_rsc(html) or
    parse_detail_dom(html)`.
  - **No Store/README/schema copy change needed.** Every field the listing
    advertises is genuinely present in the real payload and now parses —
    confirmed against real markup, not inferred. `epc_graph_url` /
    `price_qualifier` / `price_frequency` stay null on some listings
    (Zoopla's own `hasEpc: false`, no agent-set qualifier, not a rental) —
    that is honest, conditional nullability, not a dead field, so nothing
    was dropped from `ResultRow`/the README output table/the Store
    description.
  - `tests/fixtures/detail_page_rsc.html`: REAL page markup (not
    hand-built), trimmed from `DETAIL_PARSE_MISS_HTML` to the 3 `<script>`
    tags carrying the `ListingData` object + its description chunk, plus
    the head tags the geo-splash guard checks. 6 new tests including a
    fill-rate gate (`test_parse_detail_rsc_fill_rate_on_the_real_fixture_
    is_complete`) that asserts 16 named fields come back non-null against
    that real fixture — a schema-shaped all-`None` row now fails the suite,
    not just "doesn't raise." 130/130 tests green (124 pre-existing + 6
    new), `ruff` clean, `pyright` 0 errors.

### 0.6 — 2026-09-17

- **Fix: 21% customer success persists after 0.5 (build 0.5.1, live since
  2026-09-12) — 25/32 runs failed over the trailing 30 days, 24 of those 25
  TIMED-OUT.** 0.2 through 0.5 each closed one more unbounded SDK/browser
  round-trip (the search+detail loop's total wall clock, `open_page`'s
  `new_context`/`new_page`, every post-loop Actor SDK call, Camoufox's own
  launch/teardown, and finally `page.content()`), but all five versions
  shared the exact same defect underneath every one of those fixes:
  `src/scraper.py::RUN_BUDGET_SECONDS = 3300.0` was a hardcoded constant,
  guessed to sit safely under this Actor's platform-declared
  `defaultRunOptions.timeoutSecs` (3600s, confirmed via `GET /v2/acts/{id}`
  — note this value has never actually lived in `.actor/actor.json`, only on
  the platform side). That assumption silently breaks the moment a run
  starts with a shorter `timeoutSecs` override — Apify lets any caller
  (including this Actor's own customer, `totalUsers30Days: 1` per the 0.5
  note, i.e. one integration calling the API directly) pass a shorter
  timeout per run. A hardcoded 3300s budget then never trips before the
  platform's own, shorter kill clock does — the run hangs to the real
  deadline and gets force-killed TIMED-OUT, indistinguishable from every
  prior round's failure signature. This is the identical root cause found
  and fixed the same day on `leboncoin-france-cars` (0.7.1, commits
  `65cccf85`/`416170b9`): bounding every individual round-trip is not the
  same as bounding the run against its REAL deadline.
  - Fix: `src/main.py::_resolve_run_budget()` reads
    `Actor.configuration.timeout_at` (the platform's actual, per-run kill
    clock) minus a new `RUN_BUDGET_MARGIN_S = 300.0` safety margin, and
    threads that real budget into `scraper.scrape()` via a new optional
    `budget` parameter on `_run_scrape()`. Falls back to the old
    `RUN_BUDGET_SECONDS` constant only when `timeout_at` is unavailable
    (local `apify run`, which has no platform-assigned deadline).
  - `src/scraper.py::RunBudget` now carries its own `budget_seconds` field
    (default unchanged at `RUN_BUDGET_SECONDS`) instead of reading the
    module constant directly inside `exceeded()`, and `scrape()` /
    `RunBudget.start()` accept an externally-supplied budget — fully
    backward compatible with every existing standalone/test call site that
    doesn't have a real platform deadline to thread through.
  - **Charge-order gate checked and already correct** (unlike leboncoin,
    which needed this fixed today): `main()` already calls
    `_parse_input_or_die()` before `Actor.charge("actor-start")` fires, so a
    customer with a malformed input has never been billed here.
  - 9 new tests in `tests/test_main.py`
    (`_resolve_run_budget`: real `timeout_at`, missing-`timeout_at`
    fallback, near-zero-remaining clamp, and the exact bug this fix closes
    — a customer `timeoutSecs` override shorter than the old hardcoded
    3300s constant; plus a test proving `_run_scrape()` actually forwards
    the real budget object into `scrape()`, not just constructs it unused).
    124/124 total tests green (115 pre-existing + 9 new), `ruff check`
    clean, `pyright` 0 errors, `verify_input_prefill.py` and
    `verify_no_scaffold_stub.py` both OK.

### 0.5 — 2026-09-12

- **Fix: 22% customer success persists after 0.4 (build 0.4.1, live since
  2026-09-09) — 20/21 runs TIMED-OUT over the trailing 30 days
  (`publicActorRunStats30Days`), last customer run 2026-09-11.** 0.2, 0.3,
  and 0.4 each bounded one more unbounded SDK/browser round-trip (the
  search+detail loop's total wall clock, `open_page`'s
  `new_context`/`new_page`, every post-loop Actor SDK call, and Camoufox's
  own launch/teardown) but all three missed a call sitting right next to an
  already-bounded one in the exact same function: `src/browser.py::fetch_html()`
  calls `page.goto(..., timeout=GOTO_TIMEOUT_MS)` (bounded) immediately
  followed by `page.content()` (**not** bounded). Unlike every other
  Playwright `Page` method used in this codebase, `Page.content()` takes no
  `timeout` argument at all and is not covered by
  `page.set_default_timeout()` either (confirmed against the installed
  `playwright` package's generated API) — so a page left unresponsive after
  `domcontentloaded` (a stuck anti-bot challenge, a wedged Juggler/CDP
  round-trip) hangs this call forever, with nothing standing between it and
  the platform's declared run timeout. `fetch_html` is the busiest single
  call in the run: once per search fetch AND once per every detail fetch
  (up to `maxProperties`), so any one page going unresponsive after
  `domcontentloaded` could hang the whole run — indistinguishable from a
  TIMED-OUT run's own dashboard entry, with no explanation in the log.
  - **Shared root cause, not shared fix:** the identical unbounded
    `page.content()` call (no timeout, same fetch-then-return-html shape)
    exists in the sibling Camoufox actor `cars-com-listings-scraper`
    (`src/browser.py::fetch_page`, line ~159), which shows the same 100%
    TIMED-OUT signature right now — one bug in a shared fetch-page pattern
    across two actors, not two independent defects. That actor is out of
    scope for this fix (a separate fixer owns it) and was not touched here.
  - Fix: new `CONTENT_TIMEOUT_S = 20.0` wraps `page.content()` in
    `asyncio.wait_for`. A stall raises stdlib `TimeoutError`, already a
    member of `RECOVERABLE_BROWSER_ERRORS` since 0.3, so it degrades
    through `fetch_html`'s existing `except RECOVERABLE_BROWSER_ERRORS`
    clause (`return None`) — a stalled search fetch is treated as a
    geo-splash-guard failure (session rotate + retry) and a stalled detail
    fetch degrades that one listing to its search-card fields, exactly
    mirroring how every other bounded call in this module already degrades.
  - **Not yet separately confirmed as THE complete explanation**: as with
    every prior round, `/v2/acts/{id}/runs` only exposes this Actor's own 9
    runs (all SUCCEEDED/FAILED, none TIMED-OUT — all pre-0.5) because the 20
    failing customer runs execute under the customer's own account and
    their logs are unreadable to this token
    (`reference-ppe-customer-runs-bill-us`). This fix closes the one
    remaining unbounded-wait gap a full audit of every `await` in the
    request path found; a live 30-day window after 0.5 ships is the only
    way to confirm no fifth stall site remains. Also notable:
    `publicActorRunStats30Days.totalUsers30Days` is `1` — all 27 runs in the
    window belong to a single customer/integration, not a broad customer
    base, which does not change the fix but is worth flagging for the next
    fixer if the pattern recurs.
  - 2 new tests in `tests/test_browser.py`
    (`test_fetch_html_returns_none_when_content_stalls`,
    `test_fetch_html_content_timeout_does_not_block_the_event_loop`), 119
    total green. `ruff` clean, `pyright` 0 errors.

### 0.4 — 2026-09-09

- **Fix: 11% customer success persists after 0.3 (build 0.3.1, live since
  2026-09-01) — 17/19 runs TIMED-OUT over the trailing 30 days, last
  customer run 2026-09-08.** 0.3 bounded two unbounded SDK/browser
  round-trips (`open_page`'s `new_context`/`new_page`, and every
  post-loop Actor SDK call in `main.py`) but missed a THIRD: Camoufox's
  own process launch and teardown.
  - `src/browser.py::open_browser()` called `AsyncCamoufox.__aenter__()`
    (the actual browser process launch) with no timeout of its own,
    once per geo-splash-guard retry attempt in `scraper.py::_try_once`
    — **before** `scraper.RunBudget` is ever checked. A wedged launch
    (an under-loaded or slow Camoufox process start, more likely under
    sustained anti-bot pressure on a residential proxy exit) hangs the
    entire run past the platform's declared `timeoutSecs` with nothing
    to stop it — exactly the TIMED-OUT signature that persisted through
    the whole 0.3.1 measurement window.
  - `_close_browser_quietly()`'s `AsyncCamoufox.__aexit__()` (process
    teardown, called at the end of every attempt, including a
    successful one) had the same gap — a wedged close could strand an
    otherwise-complete, fully-scraped run the same way.
  - Fix: new `BROWSER_LAUNCH_TIMEOUT_S = 60.0` wraps `__aenter__` in
    `asyncio.wait_for`; a stall raises stdlib `TimeoutError`, already a
    member of `RECOVERABLE_BROWSER_ERRORS` since 0.3, so it degrades
    through the existing `_try_once` except-clause (rotate session,
    retry) instead of propagating uncaught. New
    `BROWSER_CLOSE_TIMEOUT_S = 30.0` wraps `__aexit__`; a stall there is
    swallowed-and-logged (mirrors `main.py`'s `_set_status_message`/
    `_set_value` guards — a cosmetic teardown must never hold
    already-built rows hostage).
  - **Not yet separately confirmed as THE root cause**: `/v2/acts/{id}/runs`
    only exposes our own 8 runs (all SUCCEEDED, all on the 5-row QA
    fixture) — the 17 failing customer runs execute under the
    customer's own account and their logs are unreadable to this
    token. This fix closes the one remaining unbounded-wait gap this
    fleet's proven pattern (`reference-fleet-fault-isolation-pattern`)
    predicts, verified against the live target locally, but a live
    30-day window after this ships is the only way to confirm no
    fourth stall site remains.
  - 117 tests green (2 new: `test_open_browser_raises_timeout_error_on_a_wedged_launch`,
    `test_open_browser_teardown_swallows_a_stall` in `tests/test_browser.py`),
    `ruff` clean, `pyright` 0 errors, `verify_input_prefill` /
    `verify_no_scaffold_stub` both OK.

### 0.3 — 2026-09-01

- **Fix: 18% customer success persists after 0.2, 9/9 failures TIMED-OUT.**
  0.2's `RunBudget` correctly bounds the search+detail loop's total wall
  clock (3300s), but two unbounded SDK/browser round-trips remained —
  the same defect shape that hit `kick-chat-archive` (0.15) and
  `vrbo-vacation-rentals-scraper` (0.4) the same day:
  1. `src/browser.py::open_page()` — neither Playwright's
     `browser.new_context()` nor `context.new_page()` has a timeout of
     its own, unlike `fetch_html()`'s `page.goto()`
     (`GOTO_TIMEOUT_MS`). A wedged Camoufox process could hang either
     call indefinitely. Bounded with a new `PAGE_OPEN_TIMEOUT_S = 30.0`
     via `asyncio.wait_for`, and added stdlib `TimeoutError` to
     `RECOVERABLE_BROWSER_ERRORS` so the bound firing degrades one
     listing/search-fetch through the existing per-call
     `except RECOVERABLE_BROWSER_ERRORS` handlers in `scraper.py`,
     rather than propagating uncaught.
  2. `src/main.py` — every Actor SDK round-trip AFTER the
     budget-bounded loop (`Actor.push_data`, `Actor.charge`,
     `Actor.set_status_message`, `Actor.set_value`) had no timeout of
     its own and shares apify_client's own retry policy (confirmed
     elsewhere in the fleet to stall up to ~48 minutes under a
     platform-side hiccup). With `RUN_BUDGET_SECONDS = 3300` and this
     Actor's declared `defaultRunOptions.timeoutSecs: 3600`, only ~300s
     of headroom remained for these calls — a single stall on any one
     of them turns a functionally-complete run (rows built, ready to
     push) into a platform-forced TIMED-OUT. Added `PUSH_TIMEOUT_S =
     60.0` and wrapped every call site: `push_data` now raises a new
     `PushStalledError` on stall (fails fast and loud instead of
     reporting a false "Done" for data that never landed — never
     swallowed, unlike the others); `charge` swallows-and-logs
     (unchanged non-fatal PPE contract, now bounded); the three
     `set_status_message` call sites and two `set_value` call sites in
     `_finalize`/`_dump_empty_search_debug`/`_dump_detail_parse_miss_debug`
     now go through new guarded wrappers `_set_status_message`/
     `_set_value` that swallow-and-log a stall — a cosmetic status
     write or diagnostic dump must never hold already-landed data or
     charges hostage.
  - Own-account runs never reproduced either gap: QA/local runs use a
    tiny fixture that never approaches the wall-clock or SDK-retry
    conditions needed to trigger them, and `/v2/acts/{id}/runs` cannot
    see the failing customer runs at all (PPE customer runs execute
    under the customer's own account) — confirmed via the REST API
    that our own run history only holds 7 runs, all SUCCEEDED or
    pre-0.2, so the 9 TIMED-OUT failures behind this fix are diagnosed
    from code audit against the proven fleet-wide pattern, not a
    reproduced log.
  - Version 0.2 → 0.3. 10 new tests across `tests/test_main.py`
    (`PushStalledError`, `_charge`/`_set_status_message`/`_set_value`
    stall-and-swallow/succeed pairs) and `tests/test_browser.py`
    (`open_page` timeout + `TimeoutError` membership in
    `RECOVERABLE_BROWSER_ERRORS`).

### 0.2 — 2026-08-25

- **Fix: 0% customer success, 5/5 runs TIMED-OUT (30-day public stats).**
  Root cause is `reference-fleet-fault-isolation-pattern`, the fleet's #1
  recurring bug (previously hit `workday-jobs-scraper` for the identical
  reason): every individual page fetch already had its own bounded
  timeout (`browser.GOTO_TIMEOUT_MS`, 45s), but nothing capped the SUM
  across many listings. With `enrichDetails=true` (the default) and a
  degraded/slow proxy session, up to `maxProperties` (schema max 500;
  default 50) sequential detail-page fetches could each burn close to
  their full timeout, walking the total run time past the Actor's own
  declared `defaultRunOptions.timeoutSecs: 3600` and getting hard-killed
  by the platform as `TIMED-OUT` instead of finishing with a partial
  dataset. Own-account QA runs never reproduced it because the QA
  fixture (`maxProperties: 5`) never accumulated enough sequential
  detail-page time to approach the 1-hour ceiling — this is exactly why
  the daily fleet-health signal (customer runs) caught it and QA didn't.
  - Added `src/scraper.py::RunBudget` — a wall-clock budget
    (`RUN_BUDGET_SECONDS = 3300`, 55 min, comfortably under the
    declared 3600s) shared across every geo-splash-guard retry in one
    `scrape()` call (a retry does not reset the clock). `clock` is
    injectable for tests.
  - `_build_rows()` checks the budget before each listing's detail
    fetch and stops early once it's expired — the un-processed tail is
    counted in `ScrapeOutcome.listings_skipped` (not silently dropped)
    and `ScrapeOutcome.budget_exhausted` is set so the run reports a
    clear partial-success status message
    (`main.py::_finalize` — "... stopped early: run-time budget
    reached") instead of running out the clock.
  - `scrape()`'s outer geo-splash-guard retry loop also checks the
    budget between attempts so it doesn't sleep+retry pointlessly once
    the run is already out of time.
  - This never changes the zero-rows fail-loud contract (ADR-0002 §6):
    if the budget expires before any listing is built, the run still
    fails loud with zero rows — only a genuine partial dataset gets the
    softer partial-success path.

### 0.1 — 2026-08-20

- Fix: first cloud run (`Sfgpktp0siTm1O1Lk`, build 0.0.1) reached the
  search page and produced zero rows. Diagnosis + fix:
  - Pinned the proxy to `RESIDENTIAL` + `apifyProxyCountry: "GB"`
    (`.actor/input_schema.json`, `src/models.py` default, and
    `src/browser.py::PROXY_COUNTRY_CODE`, hardcoded — never read from
    user input, mirrors the 99acres/funda single-market pattern). The
    account is now on the STARTER plan; the old FREE-tier
    `RESIDENTIAL availableCount: 0` reasoning that justified an
    unpinned default no longer applies, and a geo-random exit was the
    likely cause of the empty/mismatched search page.
  - Added a zero-rows diagnostic: the last-fetched search page's raw
    HTML is now captured in a `debug_sink` threaded through
    `scraper.scrape()` → `_try_once()` → `_fetch_search_hits()`, and
    dumped to the KVS key `EMPTY_SEARCH_PAGE_HTML` by
    `main._dump_empty_search_debug()` before the run fails loud —
    mirrors `funda-netherlands-real-estate-scraper`'s
    `EMPTY_SEARCH_DEBUG_KEY` pattern so a block vs. a parse-miss is
    diagnosable from the KVS instead of guessed at.
- Kept the existing no-`geoip=True` / static-`locale` Camoufox launch
  pattern (proven fix, see `src/browser.py` module docstring and the
  `ai-overview-citations`/`vrbo-vacation-rentals-scraper` precedent) —
  pinning the proxy's own exit country is the safe way to fix the
  locale/exit-geo mismatch the `geoip=True` warning flags, without
  reintroducing the diagnosed pre-launch IP-echo-sweep crash risk.
- Root cause of the zero rows, confirmed via the diagnostic above (cloud
  run `YmaJV5GMdj75QRo8O`, real search-page HTML pulled from KVS): the
  proxy fix cleared Cloudflare fine (a genuine 628 KB Zoopla page came
  back, no challenge markers), but the search-card parser was wrong.
  Zoopla now serves the Next.js **App Router** — there is no
  `<script id="__NEXT_DATA__">` blob (RSC streaming instead), and the
  DOM shape the original hand-guessed selectors targeted
  (`[data-testid="search-result"]`, `data-listing-id`, etc.) does not
  exist on the live site. Rewrote `src/parsers/search.py::parse_search_dom`
  against the real shape: cards live in
  `[data-testid="regular-listings"] div[id^="listing_"]`, price/beds/
  baths/sqft/tenure are extracted from the row's own text via regex
  (CSS module class names are hashed per build and not selected on),
  address comes from the semantic `<address>` tag, and listing date
  from `<time datetime>`. `parse_search_json` stays as a defensive
  first try for a hypothetical legacy page. `tests/fixtures/search_page.html`
  replaced with a fixture mirroring the confirmed real shape.
- Added a second, unconditional diagnostic
  (`main._dump_detail_parse_miss_debug`, KVS key `DETAIL_PARSE_MISS_HTML`)
  that captures the first unparseable detail page in a run — detail-page
  DOM has not been live-recon'd yet (still hand-built/spec-guessed), so
  `enrichDetails=True` rows may currently degrade to search-card-only
  fields (tracked gap, not a blocker: fault-isolated per
  `reference-fleet-fault-isolation-pattern`, never fails the run).

### 0.0 — 2026-08-13

- Scaffolded skeleton (T01). Not yet a working scraper — see
  `docs/specs/zoopla-uk-property-scraper/tasks.md` for the build order
  (T03 models, T04 Camoufox browser infra, T05 parser, T06 scraper
  orchestration, T07 real `main.py` are the remaining hard gates).