# Changelog of VRBO Vacation Rentals Scraper (`devilscrapes/vrbo-vacation-rentals-scraper`) Actor - **URL**: https://apify.com/devilscrapes/vrbo-vacation-rentals-scraper/changelog.md - **Full Actor documentation**: https://apify.com/devilscrapes/vrbo-vacation-rentals-scraper.md ## VRBO Vacation Rentals Scraper — Changelog ### 0.5 — 2026-09-14 Fleet report flagged this Actor UNDER_MAINTENANCE — 0.0% 30-day customer success (0/15 runs). Live build 0.4.1 matched `main`, so there was no undeployed fix; live-run logs (`vHQXCWSQxL26jeNiG`, `KaazHERiIGnpG0U0B`, and the whole 0.2–0.4 history back to `srEJ8FUMvOPsNnndn`) show this target's Akamai "Bot or Not?" 429 challenge is a real, recurring anti-bot wall on Apify's shared `RESIDENTIAL` pool that 0.1–0.4 already spent three release cycles retrying around (backoff, fresh-page retry, hydration-timeout headroom) without moving the customer success rate off 0%. Two independent fixes, both required: 1. **The `actor-start` PPE fee was charged unconditionally at Actor boot** — before proxy resolution, before Camoufox launched, before a single byte reached vrbo.com. Every one of those 15 failing customer runs was therefore billed for a run that delivered nothing — the exact "billed for a run that never reached the target" shape `google-ads-transparency`'s own `_StartFee` class names as *"the shape that got vrbo-vacation-rentals-scraper delisted at 17 runs"* (its own docstring, written 2026-09-13, one day before this fix). `src/main.py` now carries the same `_StartFee` pattern: `actor-start` charges at most once, and only from `_push_and_charge` the first time a destination actually delivers rows. A proxy-resolution failure or a run that never clears the Akamai wall now costs the customer nothing, and `_finalize`'s zero-row status message says so explicitly ("nothing was charged"). 2. **`WEBSHARE_PROXY_URL` — the env var 0.1's fix depends on to route around the Akamai-flagged `RESIDENTIAL` pool — was silently absent from the live 0.4 version.** `GET .../versions` confirms it: versions 0.1–0.3 each carry a `WEBSHARE_PROXY_URL` secret entry, 0.4 carries none (the exact `apify push`-drops-secret-env-vars gotcha this fleet has hit before). Log evidence lines up: every 0.4.1 customer run logs `"WEBSHARE_PROXY_URL not set; falling back to Apify Proxy"` and then a 429 "Bot or Not?" 2–3 attempts later. `.actor/actor.json` now declares `"environmentVariables": {"WEBSHARE_PROXY_URL": "@WEBSHARE_PROXY_URL"}` — `apify push` resolves the leading `@` against the local `apify secrets` store (confirmed against apify-cli 1.6.1's own `isSecretKey`/`replaceSecretsValue`: the docs' `@secret:NAME` form is NOT what this CLI version parses — it strips only the leading `@` and looks up the remainder verbatim) — so the secret ships as part of the *source*, not a Console side-channel, and every future `apify push` carries it forward instead of silently dropping it again. Per-destination fault isolation (a search-fetch crash skips one destination, never the whole run) and the Akamai retry/backoff loop were already correct from 0.3/0.4 and are unchanged here. 9 new/changed regression tests in `tests/test_main.py`: `_StartFee` charge-once semantics, `_push_and_charge` gating actor-start on a delivered batch (and never charging it on an empty batch), `_LoopContext`'s default `start_fee`, and two `main()`-level tests — one proving a proxy-resolution failure charges nothing, one proving a delivering run still bills exactly one `actor-start`. Full local suite: 121/121 green (1 smoke test deselected — network). `ruff check`/`ruff format --check`/`pyright` all clean. `scripts/verify_charge_order.py` now reports `no-start-charge` for this Actor (no unconditional boot-time charge to check the ordering of). ### 0.4 — 2026-09-01 Fleet report flagged this Actor at 0% again (16/16 customer runs failed, 6 TIMED-OUT, 30-day window) even though 0.3's build (deployed 2026-08-26) had already cleared cloud QA. `recent_failures` (our own runs) were all pre-0.3.2, so the count was mostly stale window data — but the six TIMED-OUT customer runs, unseen in our own logs, could not be explained away without a code-level look at what's still unbounded. Two independent gaps found and fixed: 1. **Missing per-item deadlines** (`src/browser.py`): `open_page` (`browser.new_context`/`new_page`) and `_capture_one`'s `response.body()` read carried no timeout of their own — unlike `page.goto`/`wait_for_search_results`, which already pass explicit `timeout=`. Either can block indefinitely against a wedged Camoufox process or a target holding a `/graphql` connection open mid-stream. `open_page` is called dozens of times per run; a single hang there had nothing standing between it and the platform's own kill — TIMED-OUT instead of a clean FAILED/partial finalize. Both are now bounded (`PAGE_OPEN_TIMEOUT_S=30`, `GRAPHQL_BODY_READ_TIMEOUT_S=15`), and stdlib `TimeoutError` was added to `RECOVERABLE_BROWSER_ERRORS` so a bound firing degrades the one listing/destination in flight via the existing per-item fault-isolation handlers instead of only being caught (mis-attributed as "hard deadline exceeded") by `main.py`'s outer hard-deadline catch. 2. **Retry loop gave up too early on a silent soft-throttle** (`src/search_fetch.py`): live recon today reproduced a real run clearing an explicit 429 "Bot or Not?" on attempt 1, then landing a `status=200` page with no challenge markers on attempt 2 that never hydrated within the timeout and delivered zero GraphQL/DOM data. The old retry predicate only fired on an explicit marker match, so it treated that as a genuine empty result and returned immediately, using only 1 of 3 available attempts. A hydration timeout is now retry-worthy on its own — only a page that *did* hydrate and is still empty is treated as a real no-match (REQ-8 unchanged). `src/graphql_capture.py` is a new module — GraphQL capture + hydration-wait code split out of `browser.py`, which had crossed the 400-line ceiling (ADR-0003) once the new timeout bound was added; this is also a genuinely distinct concern from browser/page lifecycle. Tests moved to `tests/test_graphql_capture.py` alongside it. 9 new/changed regression tests: `test_recoverable_errors_include_stdlib_timeout_error`, `test_open_page_returns_the_new_page_on_the_happy_path`, `test_open_page_raises_timeout_error_instead_of_hanging_forever`, `test_capture_graphql_responses_bounds_a_hanging_body_read`, `test_fetch_search_hits_retries_on_hydration_timeout_alone` (rewritten from the old single-attempt expectation), plus the moved/rewritten `test_graphql_capture.py` suite. Full local suite: 113/113 green. `ruff`/`pyright` clean on all touched files. Live recon also surfaced target-side flakiness independent of this fix — a hydrated, marker-free page sometimes still returns zero GraphQL/DOM hits (observed twice today); not reproduced deterministically enough to root cause or fix here. Flagged in notes.md for a future investigation. ### 0.3 — 2026-08-26 Fixes the 0.2 regression's real cause: customer success stayed at 0% (10/10 FAILED, 3 TIMED-OUT) even after the timeout/deadline fix, because the actor had **zero retry** on an Akamai "Bot or Not?" 429 challenge — a gap already flagged in the 0.1 notes but never implemented. All 3 most recent cloud runs (`qKVVlgycZLKaCqjKf`, `gsXiUaRyEuQi8xfiR` on 0.2.1; `yo51UTRl1iC446MOe` on 0.2.2) failed on exactly this: one navigation attempt, a 429 challenge, zero hits, fail loud — correct behavior on a genuine block, but nothing tried to get past a *transient* one. Recon before writing the fix (`actors/vrbo-vacation-rentals-scraper/.venv`, live Webshare proxy): 3/3 fresh Camoufox browsers cleared the same search URL clean (`status=200`, no challenge) — the block isn't durable. A follow-up probe showed a fresh `new_context()`/`new_page()` cycle on the *same* Camoufox browser already draws a different exit IP through Webshare's rotating gateway (confirmed via `api.ipify.org`) — so a retry doesn't need a full browser relaunch, just a fresh page. - `src/search_fetch.py` (new module, split out of `main.py` to stay under the 400-line ceiling — ADR-0003): `fetch_search_hits` now retries up to `SEARCH_CHALLENGE_MAX_ATTEMPTS` (3) times on a detected challenge, opening a fresh page each attempt, with exponential backoff (`SEARCH_CHALLENGE_BACKOFF_BASE_S` doubling, capped at `SEARCH_CHALLENGE_BACKOFF_CAP_S`). A genuinely-empty **non-challenged** page (real no-match) is *not* retried — REQ-8's "empty is not failure" rule is unchanged, only a detected block earns a second try. - `GOTO_TIMEOUT_MS` moved to `src/browser.py` so both `main.py` (detail fetch) and `search_fetch.py` (search fetch) share one definition. - Verified live locally (`apify run` against `tests/fixtures/input.qa.json`, real Webshare proxy, real vrbo.com): attempt 1 hit the real 429 "Bot or Not?" page, the retry cleared it on attempt 2, run **SUCCEEDED with 3 real rows** (not zero — see `reference-success-is-not-delivery`). - 3 new regression tests in `tests/test_search_fetch.py` (retry-and-succeed, exhausts-all-attempts-still-challenged, does-not-retry-genuine-empty) plus the 2 pre-existing hydration tests moved there from `test_main.py`. Added `tests/test_smoke.py` (`-m smoke`) exercising `fetch_search_hits` against the live target — closes the actor-publisher Gate 7 flag. Full local suite: 109/109 hermetic + 1/1 smoke green. `ruff`/`pyright` clean. **Cloud QA — PASS (2 billing-verified runs, build 0.3.2):** - `apify push --force` (actor-publisher, build `ky3902uDFey81TCeQ` / 0.3.1) found a real deployment gap along the way: `apify push` does **not** carry a version's secret env vars forward — `GET .../versions` showed 0.1 and 0.2 each had their own `WEBSHARE_PROXY_URL` entry, 0.3 had none. First QA run (`Irpi8fTrfarvAGQHv`) fell back to Apify's shared `RESIDENTIAL` pool and — notably — **succeeded cleanly with 3 real rows, zero challenges**, billing-confirmed real residential transfer (`run_usage.py`: `PROXY_RESIDENTIAL_TRANSFER_GBYTES=0.0235`). Fresh same-day evidence Apify's own pool isn't durably blocked either; logged as a follow-up question, not acted on here (see notes.md). - Restored `WEBSHARE_PROXY_URL` on version 0.3 via `PUT /v2/acts/{id}/versions/0.3` and rebuilt (`FyM3SsXaQJcwdYFNi` / 0.3.2) so the fix could be tested against the exact target it names. - Second QA run (`iohZfJMehAaEcjuK2`, build 0.3.2) is the real proof: log shows **two genuine 429 "Bot or Not?" challenges** (attempts 1 and 2), each retried with a fresh page and the documented 2s/4s backoff, **attempt 3 cleared it** — run SUCCEEDED, 3 real rows (`The Point Hotel & Suites Orlando`, `The Ocean's Cove`, `Premier Resort Condos Near Universal`). Billing-confirmed genuine Webshare/external path (`run_usage.py`: no residential transfer, `DATA_TRANSFER_EXTERNAL_GBYTES` present) — not a silent fallback. ### 0.2 — 2026-08-21 Fixes the 0% customer-run success rate (7/7 failed, 2 TIMED-OUT — flagged by the daily fleet report) after reproducing both failure buckets with real cloud runs (not inferred from source): - **FAILED bucket — search-hydration timeout too tight for the cloud container.** A cloud run with default-prefill customer input (runId `srEJ8FUMvOPsNnndn`) reproduced "Zero hits (graphql+dom)" on a page that had genuinely loaded (`status=200`, no Akamai challenge markers, correct title) — the search SPA simply hadn't hydrated within the old 25s ceiling. A local recon through the identical Webshare proxy hydrated the same target in ~20s, confirming this isn't a block, just insufficient headroom for the Apify container (xvfb + CPU + proxy latency). Raised `SEARCH_RESULTS_WAIT_TIMEOUT_MS` 25s → 45s. - **TIMED-OUT bucket — internal budget had no headroom under the platform timeout.** The run-wide `maxRunMinutes` budget only gates *starting* new destinations/listings; it never preempts a page op already in flight. At the old ceiling (60 min) plus the worst-case tail of one in-flight destination search + one in-flight detail fetch (~150s, more with the 45s hydration wait above), a run could spill past the platform's 3600s (60-min) default `timeoutSecs` and finish TIMED-OUT with no clean finalize. Lowered `maxRunMinutes`'s max from 60 to 45 (900s headroom) and added `_run_destinations_with_hard_deadline` — an `asyncio.wait_for` ceiling (`maxRunMinutes*60 + 240s` grace) around the whole destination loop, mirroring `workday-jobs-scraper`'s proven hard-deadline pattern (`docs/specs/workday-jobs-scraper/notes.md`). If it ever fires, the run fails loud with a clear status message instead of riding to the platform's kill; any rows already pushed by completed destinations are unaffected. - Ruled out the `threads-reply-tree`-class bug (silent fallback to the datacenter `BUYPROXIES94952` pool): this Actor prefers `WEBSHARE_PROXY_URL` unconditionally when set, confirmed live in the reproduced run's log (`"Using Webshare proxy (WEBSHARE_PROXY_URL set)"`) and via `run_usage.py` showing real non-zero `DATA_TRANSFER_EXTERNAL_GBYTES` on both the failing and a passing run — the zero `PROXY_RESIDENTIAL_TRANSFER_GBYTES` reading is expected here, not a silent downgrade. ### 0.1 — 2026-08-11 Fixes two cloud-QA blockers found after the initial Camoufox build: - **Proxy**: Apify's shared `RESIDENTIAL` proxy pool is Akamai-flagged for vrbo.com (429 "Bot or Not?" challenge, confirmed via cloud QA). `resolve_proxy` now prefers a `WEBSHARE_PROXY_URL` env var (parsed into the same Playwright-shaped proxy dict) when set, falling back to the original Apify RESIDENTIAL → BUYPROXIES94952 resolution otherwise. **`WEBSHARE_PROXY_URL` must be set as an Actor env var / secret in Apify Console for cloud runs** — it is not forwarded from local `.env`. - **Hydration**: the search-results page's `DeferredSearchResults` GraphQL response (~1.7MB) wasn't reliably landing inside the previous fixed 6s post-load wait — cloud QA captured a 133-char mid-load body ("Searching thousands of properties"). Replaced the blind `page.wait_for_timeout(...)` with `wait_for_search_results`, which polls for either a captured GraphQL response or the results-card DOM selector, capped at a 25s timeout, before falling through to the existing GraphQL-first/DOM-fallback extraction. ### 0.0 — 2026-08-11 - Scaffolded skeleton (T01). Not yet a working scraper — see `docs/specs/vrbo-vacation-rentals-scraper/tasks.md` for the build order (T04 Camoufox infra proof is the next hard gate).