VRBO Vacation Rentals Scraper
DeprecatedPricing
Pay per event
Scrape VRBO (Expedia Group) vacation-rental search results and listing details by destination: title, location, price per night, beds/baths, guests, rating, reviews, amenities, host, and coordinates. Camoufox-powered to clear anti-bot challenges. Pay only per listing scraped.
Fleet report flagged this Actor at 0% again (16/16 customer runs failed,
6 TIMED-OUT, 30-day window) even though 0.3's build (deployed 2026-08-26)
had already cleared cloud QA. recent_failures (our own runs) were all
pre-0.3.2, so the count was mostly stale window data — but the six
TIMED-OUT customer runs, unseen in our own logs, could not be explained
away without a code-level look at what's still unbounded.
Two independent gaps found and fixed:
src/browser.py): open_page
(browser.new_context/new_page) and _capture_one's
response.body() read carried no timeout of their own — unlike
page.goto/wait_for_search_results, which already pass explicit
timeout=. Either can block indefinitely against a wedged Camoufox
process or a target holding a /graphql connection open mid-stream.
open_page is called dozens of times per run; a single hang there
had nothing standing between it and the platform's own kill —
TIMED-OUT instead of a clean FAILED/partial finalize. Both are now
bounded (PAGE_OPEN_TIMEOUT_S=30, GRAPHQL_BODY_READ_TIMEOUT_S=15),
and stdlib TimeoutError was added to RECOVERABLE_BROWSER_ERRORS so
a bound firing degrades the one listing/destination in flight via the
existing per-item fault-isolation handlers instead of only being
caught (mis-attributed as "hard deadline exceeded") by main.py's
outer hard-deadline catch.src/search_fetch.py): live recon today reproduced a real run
clearing an explicit 429 "Bot or Not?" on attempt 1, then landing a
status=200 page with no challenge markers on attempt 2 that never
hydrated within the timeout and delivered zero GraphQL/DOM data. The
old retry predicate only fired on an explicit marker match, so it
treated that as a genuine empty result and returned immediately,
using only 1 of 3 available attempts. A hydration timeout is now
retry-worthy on its own — only a page that did hydrate and is
still empty is treated as a real no-match (REQ-8 unchanged).src/graphql_capture.py is a new module — GraphQL capture +
hydration-wait code split out of browser.py, which had crossed the
400-line ceiling (ADR-0003) once the new timeout bound was added; this is
also a genuinely distinct concern from browser/page lifecycle. Tests moved
to tests/test_graphql_capture.py alongside it.
9 new/changed regression tests: test_recoverable_errors_include_stdlib_timeout_error,
test_open_page_returns_the_new_page_on_the_happy_path,
test_open_page_raises_timeout_error_instead_of_hanging_forever,
test_capture_graphql_responses_bounds_a_hanging_body_read,
test_fetch_search_hits_retries_on_hydration_timeout_alone (rewritten from
the old single-attempt expectation), plus the moved/rewritten
test_graphql_capture.py suite. Full local suite: 113/113 green.
ruff/pyright clean on all touched files.
Live recon also surfaced target-side flakiness independent of this fix — a hydrated, marker-free page sometimes still returns zero GraphQL/DOM hits (observed twice today); not reproduced deterministically enough to root cause or fix here. Flagged in notes.md for a future investigation.
Fixes the 0.2 regression's real cause: customer success stayed at 0% (10/10
FAILED, 3 TIMED-OUT) even after the timeout/deadline fix, because the actor
had zero retry on an Akamai "Bot or Not?" 429 challenge — a gap already
flagged in the 0.1 notes but never implemented. All 3 most recent cloud runs
(qKVVlgycZLKaCqjKf, gsXiUaRyEuQi8xfiR on 0.2.1; yo51UTRl1iC446MOe on
0.2.2) failed on exactly this: one navigation attempt, a 429 challenge, zero
hits, fail loud — correct behavior on a genuine block, but nothing tried to
get past a transient one.
Recon before writing the fix (actors/vrbo-vacation-rentals-scraper/.venv,
live Webshare proxy): 3/3 fresh Camoufox browsers cleared the same search URL
clean (status=200, no challenge) — the block isn't durable. A follow-up
probe showed a fresh new_context()/new_page() cycle on the same
Camoufox browser already draws a different exit IP through Webshare's
rotating gateway (confirmed via api.ipify.org) — so a retry doesn't need a
full browser relaunch, just a fresh page.
src/search_fetch.py (new module, split out of main.py to stay under
the 400-line ceiling — ADR-0003): fetch_search_hits now retries up to
SEARCH_CHALLENGE_MAX_ATTEMPTS (3) times on a detected challenge, opening
a fresh page each attempt, with exponential backoff
(SEARCH_CHALLENGE_BACKOFF_BASE_S doubling, capped at
SEARCH_CHALLENGE_BACKOFF_CAP_S). A genuinely-empty non-challenged
page (real no-match) is not retried — REQ-8's "empty is not failure"
rule is unchanged, only a detected block earns a second try.GOTO_TIMEOUT_MS moved to src/browser.py so both main.py (detail
fetch) and search_fetch.py (search fetch) share one definition.apify run against tests/fixtures/input.qa.json,
real Webshare proxy, real vrbo.com): attempt 1 hit the real 429
"Bot or Not?" page, the retry cleared it on attempt 2, run SUCCEEDED
with 3 real rows (not zero — see reference-success-is-not-delivery).tests/test_search_fetch.py (retry-and-succeed,
exhausts-all-attempts-still-challenged, does-not-retry-genuine-empty) plus
the 2 pre-existing hydration tests moved there from test_main.py. Added
tests/test_smoke.py (-m smoke) exercising fetch_search_hits against
the live target — closes the actor-publisher Gate 7 flag. Full local
suite: 109/109 hermetic + 1/1 smoke green. ruff/pyright clean.Cloud QA — PASS (2 billing-verified runs, build 0.3.2):
apify push --force (actor-publisher, build ky3902uDFey81TCeQ / 0.3.1)
found a real deployment gap along the way: apify push does not
carry a version's secret env vars forward — GET .../versions showed
0.1 and 0.2 each had their own WEBSHARE_PROXY_URL entry, 0.3 had none.
First QA run (Irpi8fTrfarvAGQHv) fell back to Apify's shared
RESIDENTIAL pool and — notably — succeeded cleanly with 3 real rows,
zero challenges, billing-confirmed real residential transfer
(run_usage.py: PROXY_RESIDENTIAL_TRANSFER_GBYTES=0.0235). Fresh
same-day evidence Apify's own pool isn't durably blocked either; logged
as a follow-up question, not acted on here (see notes.md).WEBSHARE_PROXY_URL on version 0.3 via PUT /v2/acts/{id}/versions/0.3FyM3SsXaQJcwdYFNi / 0.3.2) so
the fix could be tested against the exact target it names.iohZfJMehAaEcjuK2, build 0.3.2) is the real proof: log
shows two genuine 429 "Bot or Not?" challenges (attempts 1 and 2),
each retried with a fresh page and the documented 2s/4s backoff, attempt
3 cleared it — run SUCCEEDED, 3 real rows
(The Point Hotel & Suites Orlando, The Ocean's Cove, Premier Resort Condos Near Universalrun_usage.py: no residential transfer, DATA_TRANSFER_EXTERNAL_GBYTES
present) — not a silent fallback.Fixes the 0% customer-run success rate (7/7 failed, 2 TIMED-OUT — flagged by the daily fleet report) after reproducing both failure buckets with real cloud runs (not inferred from source):
srEJ8FUMvOPsNnndn) reproduced "Zero hits (graphql+dom)" on a page that
had genuinely loaded (status=200, no Akamai challenge markers, correct
title) — the search SPA simply hadn't hydrated within the old 25s ceiling.
A local recon through the identical Webshare proxy hydrated the same
target in ~20s, confirming this isn't a block, just insufficient headroom
for the Apify container (xvfb + CPU + proxy latency). Raised
SEARCH_RESULTS_WAIT_TIMEOUT_MS 25s → 45s.maxRunMinutes budget only gates starting new
destinations/listings; it never preempts a page op already in flight.
At the old ceiling (60 min) plus the worst-case tail of one in-flight
destination search + one in-flight detail fetch (~150s, more with the
45s hydration wait above), a run could spill past the platform's 3600s
(60-min) default timeoutSecs and finish TIMED-OUT with no clean
finalize. Lowered maxRunMinutes's max from 60 to 45 (900s headroom) and
added _run_destinations_with_hard_deadline — an asyncio.wait_for
ceiling (maxRunMinutes*60 + 240s grace) around the whole destination
loop, mirroring workday-jobs-scraper's proven hard-deadline pattern
(docs/specs/workday-jobs-scraper/notes.md). If it ever fires, the run
fails loud with a clear status message instead of riding to the
platform's kill; any rows already pushed by completed destinations are
unaffected.threads-reply-tree-class bug (silent fallback to the
datacenter BUYPROXIES94952 pool): this Actor prefers WEBSHARE_PROXY_URL
unconditionally when set, confirmed live in the reproduced run's log
("Using Webshare proxy (WEBSHARE_PROXY_URL set)") and via
run_usage.py showing real non-zero DATA_TRANSFER_EXTERNAL_GBYTES on
both the failing and a passing run — the zero PROXY_RESIDENTIAL_TRANSFER_GBYTES
reading is expected here, not a silent downgrade.Fixes two cloud-QA blockers found after the initial Camoufox build:
RESIDENTIAL proxy pool is Akamai-flagged
for vrbo.com (429 "Bot or Not?" challenge, confirmed via cloud QA).
resolve_proxy now prefers a WEBSHARE_PROXY_URL env var (parsed
into the same Playwright-shaped proxy dict) when set, falling back to
the original Apify RESIDENTIAL → BUYPROXIES94952 resolution
otherwise. WEBSHARE_PROXY_URL must be set as an Actor env var /
secret in Apify Console for cloud runs — it is not forwarded from
local .env.DeferredSearchResults
GraphQL response (~1.7MB) wasn't reliably landing inside the previous
fixed 6s post-load wait — cloud QA captured a 133-char mid-load body
("Searching thousands of properties"). Replaced the blind
page.wait_for_timeout(...) with wait_for_search_results, which
polls for either a captured GraphQL response or the results-card DOM
selector, capped at a 25s timeout, before falling through to the
existing GraphQL-first/DOM-fallback extraction.docs/specs/vrbo-vacation-rentals-scraper/tasks.md for the build
order (T04 Camoufox infra proof is the next hard gate).