# Changelog of Cars.com Listings Scraper (`devilscrapes/cars-com-listings-scraper`) Actor - **URL**: https://apify.com/devilscrapes/cars-com-listings-scraper/changelog.md - **Full Actor documentation**: https://apify.com/devilscrapes/cars-com-listings-scraper.md ## Changelog ### 0.8.0 — Derive the run budget from the REAL platform deadline, not a guessed constant (14/17 public runs failed, 13 TIMED-OUT) - **Root cause**: same defect class root-caused and fixed the same day on `leboncoin-france-cars` (0.7.1) -- an earlier fix (0.7, below) bounded each individual fetch's own retry loop (`FETCH_WALL_CLOCK_TIMEOUT_S=240s`) and the run-level pagination/enrichment governor, but `RUN_DEADLINE_S=1800.0` was a fixed guessed constant applied regardless of the run's ACTUAL platform-assigned timeout. `defaultRunOptions.timeoutSecs=3600` is only the *default* -- a caller can set any `timeout` on a run, and this Actor's own live build already proved it: cloud run `rIIgBlnfdwqWn8pXg` (0.6.1) TIMED-OUT at exactly its own `options.timeoutSecs=600`, far below the guessed 1800s budget that never got a chance to stop the run first. - **Second, independent gap found while auditing this**: `browser. resolve_proxy_url` (`ProxyConfiguration.new_url()`) had NO timeout of its own -- an Apify SDK round-trip that apify\_client's own retry policy can stall for tens of minutes (same class already fixed fleet-wide on `leboncoin-france-cars`, `workday-jobs-scraper`, `zoopla-uk-property- scraper`). Worse, the call sat OUTSIDE `browser_fetch.py`'s per-attempt try/except, so a single stall there bypassed EVERY wall-clock cap this Actor already had -- `FETCH_WALL_CLOCK_TIMEOUT_S`, the run-wide governor, all of it -- because the retry loop was blocked awaiting this one call, never looping to check either deadline. - **Fixed**: - `src/run_budget.py` (new): a pure `RunBudget` dataclass -- `deadline` + latching `exceeded` flag -- mirrors `leboncoin-france-cars/src/ run_budget.py` (same fix shape, same fleet audit). - `src/main.py`: `_resolve_run_budget()` derives the deadline from the REAL platform-assigned kill clock (`Actor.configuration.timeout_at`) minus a 300s safety margin (covers one already-in-flight fetch's worst case plus final flush/charge/status-message overhead), not a guessed constant -- so it tracks the run's actual `timeoutSecs`, default or caller-overridden. Falls back to 1800s when unavailable (local `apify run`). The budget is threaded into `scraper.run`/`_iter_listings`, overriding the module's own `RUN_DEADLINE_S`-derived default when supplied. `TRUNCATED_STATUS` no longer quotes a fixed minute figure -- the budget is now per-run, so no single constant describes every run. - `src/browser.py`: `resolve_proxy_url` now wraps `new_url()` in `asyncio.wait_for(timeout=PROXY_ROTATE_TIMEOUT_S=30s)`, raising the builtin `TimeoutError` (already a member of `RECOVERABLE_BROWSER_ERRORS`) on stall instead of hanging. - `src/browser_fetch.py`: moved the `resolve_proxy_url` call INSIDE `_attempt()`'s try block so a stalled rotation is caught and classified as a transport fault (fresh exit, retry) like every other recoverable error, instead of propagating uncaught and crashing the whole run. - Verified: input validation already ran BEFORE the `actor-start` PPE charge in this Actor (unlike `leboncoin-france-cars` pre-fix) -- no charge-order change was needed here. - 8 new regression tests (`test_run_budget.py`, `_resolve_run_budget` coverage in `test_main.py`, the proxy-stall-is-a-transport-fault tests in `test_browser.py`/`test_browser_fetch.py`, and the externally-supplied- budget tests in `test_scraper.py`). 150 tests green, ruff clean, pyright 0 errors. ### 0.7 — Cap the WHOLE fetch() retry loop's wall-clock time, not just each call (14/14 public runs actionable, 13 TIMED-OUT) - **Root cause, proven from our own runs on the live 0.6.1 build**: two verification runs started 9 seconds apart against the identical QA fixture right after build 0.6.1 finished (2026-09-12T09:06:52Z) came back with DIFFERENT terminal states -- `x73KS7SzTgnrM1rYs` FAILED loud via `browser_fetch.BrowserFetchError`'s attempt-count budget in ~8 minutes (the CORRECT outcome 0.4-0.6 exist to produce), while `rIIgBlnfdwqWn8pXg` TIMED-OUT at the EXACT platform-imposed `options.timeoutSecs=600` of that run (`599.824s` per `stats.runTimeSecs`) -- not at `RUN_DEADLINE_S=1800`, not at any attempt-count exhaustion. That is the tell: nothing in our own code ever gave up on that run: it was still legitimately retrying, one attempt at a time, when something OUTSIDE our code (the platform's own timeout) killed it. `scraper.py`'s `_RunGovernor.deadline_passed()` is only checked BETWEEN calls into `browser_fetch.py` (before search page N>1, before each VDP enrich) -- it never bounds a single `fetch()` call already in flight, and page 1's fetch is REQ-1-exempt from that check entirely (always attempted, by design, so a zero-row run can fail loud after at least one real try). Each retry attempt inside `fetch()` can legitimately take up to ~180s while respecting every individual Playwright-level timeout already added in 0.6 (browser launch, page open, two `page.goto`, content-ready wait, `page.content()`) -- and the attempt-count budgets alone (`BOT_WALL_MAX_ATTEMPTS=4` + `TRANSPORT_FAULT_MAX_ATTEMPTS=6` = up to 10 attempts) put NO ceiling on how long all ten together can take against a target with a measured \~50% per-attempt block rate (`CLOUD-RECON-RESULT.md`). That is the fifth instance of the "one unbounded thing" defect class 0.6's own CHANGELOG entry found four of -- this time the unbounded thing is the retry LOOP's total duration, not any single RPC inside it. - **Fixed**: `BrowserFetcher.fetch()` now also caps its own total elapsed time at `FETCH_WALL_CLOCK_TIMEOUT_S=240s`, independent of how many attempts remain in either budget, and raises `BrowserFetchError` naming which limit hit first ("wall-clock budget" vs "independent retry budgets"). Every `fetch()` call -- including the always-attempted page 1 -- now finishes loud within bounded time (≤ one attempt's worst case past the 240s mark, so ≤ ~420s) instead of still legitimately retrying when the platform, not our code, ends the run. - 1 new regression test (`test_fetch_stops_on_wall_clock_budget_before_exhausting_attempt_ budget`) drives a fake `time.monotonic()` clock to prove the wall-clock cap fires after 2 transport attempts, well short of the 6-attempt transport budget, when every attempt is slow-but-not-instantly-failing \-- the exact shape that produced `rIIgBlnfdwqWn8pXg`'s TIMED-OUT. 138 tests green, ruff clean, pyright clean. - **Honesty note -- unverified**: this is **local-green only**. The Apify account usage cap was at $97.72/$100 (resets 2026-09-18) at the time of this fix, so no `apify push`, no cloud QA, and no `apify call` ran in this pass per explicit instruction -- the orchestrator pushes and cloud-QAs this once the cap resets. This fix changes ONLY how fast a losing fetch gives up; it does not and cannot change cars.com's underlying ~50% per-attempt block rate (independently measured, `CLOUD-RECON-RESULT.md`) -- a run that clears enough attempts inside the new 240s-per-fetch ceiling still succeeds with real rows exactly as before, and a run that doesn't now fails loud, fast, and honestly instead of riding out a customer-visible, still-billed TIMED-OUT. ### 0.6 — Reconcile git with the live 0.5 deadline governor + bound every remaining unbounded await (12/12 public TIMED-OUT) - **Repo/platform drift found while triaging this**: the Apify Store already has a build `0.5.2` (pushed 2026-09-09, `apify push` from a working tree that was never committed) implementing a wall-clock `_RunGovernor` — but neither this branch nor `main` had that code in git history at all; `actor.json` here still read `0.4` and `src/ scraper.py` had no governor. The only trace was a "0.5" CHANGELOG entry sitting **uncommitted** in the shared checkout, describing a fix whose code existed only on the Apify platform. This release pulls the live `0.5.2` source (`apify pull`) and commits it verbatim (`scraper. _RunGovernor`/`RunOutcome`, `RUN_DEADLINE_S=1800`), closing that gap, then adds three more fixes found while verifying it end-to-end that the live build still doesn't have. - **Root cause (why 12/12 customer runs are still TIMED-OUT even 3 days after 0.5.2)**: the VDP circuit breaker only trips on CONSECUTIVE enrichment failures, so a target that fails roughly every OTHER attempt never trips it, and each failed attempt still pays its own bounded-but-real per-fetch retry cost — this is what 0.5.2's governor fixes, by capping TOTAL run time instead of failure streak shape. But three further gaps mean a single call can still hang with NO bound at all, which the governor cannot help with because the run never gets back to the loop where the deadline is checked: `AsyncCamoufox`'s own `__aenter__`/`__aexit__` (browser launch/teardown), `browser. new_context`/`context.new_page`, and `page.content()` all have no timeout of their own (unlike `page.goto`) — the last one confirmed live in cloud QA run `x73KS7SzTgnrM1rYs` (2026-09-12, `Page.content: Target page, context or browser has been closed`) and independently flagged the same day by the `zoopla-uk-property-scraper` hang investigation as the identical unbounded call in the identical function shape. Every `Actor.push_data`/`Actor.charge`/`Actor.create_proxy_configuration`/ `Actor.set_status_message` call in `main.py` was unbounded too, sharing `apify_client`'s own HTTP layer/retry policy — the same stall class already confirmed on `kick-chat-archive`. - **Fixed (adopted from the live 0.5.2 build, now in git)**: `scraper. _RunGovernor` carries a wall-clock `deadline_at` (`RUN_DEADLINE_S=1800`, 30 minutes) plus the existing VDP consecutive-failure breaker. `_iter_listings` checks `deadline_passed()` before every search page after page 1 (page 1 always attempted, REQ-1), and `_emit_page` checks it before every VDP enrichment attempt. `RunOutcome.truncated_by_ deadline` lets `main.py` report a truncated run as an honest partial `Done (stopped early)`, never a silent `Done`. - **Fixed (new — unbounded browser launch + page-open)**: `browser. open_browser` now bounds Camoufox's `__aenter__`/`__aexit__` with `BROWSER_LAUNCH_TIMEOUT_S=60s`/`BROWSER_CLOSE_TIMEOUT_S=15s`; `open_page`'s `new_context`/`new_page` bounded by `PAGE_OPEN_TIMEOUT_S=30s` the same way. `TimeoutError` added to `RECOVERABLE_BROWSER_ERRORS` so a launch timeout is retried as a transport fault (fresh exit), not a crash. Matches the fix already proven on `zoopla-uk-property-scraper` for the identical defect class. - **Fixed (new — unbounded `page.content()`)**: `fetch_page`'s final `html = await page.content()` now runs behind `asyncio.wait_for(..., timeout=CONTENT_TIMEOUT_S=20s)`. Same `RECOVERABLE_BROWSER_ERRORS` classification applies, so a stall here is retried as a transport fault, not a crash. - **Fixed (new — unbounded Apify SDK round-trips)**: `main.py`'s `push_data`/`charge`/`create_proxy_configuration`/`set_status_message` calls now all run behind `asyncio.wait_for(..., timeout= PUSH_TIMEOUT_S=60s)`, matching the established `fiverr-gig-listings- scraper`/`zoopla-uk-property-scraper` pattern. A stalled `push_data` now raises loud (`RuntimeError`) instead of silently reporting "Done" with rows unlanded; a stalled `charge`/`set_status_message` is swallowed (never holds already-landed data hostage). - 8 new regression tests (deadline-governor pagination/enrichment stop, browser-launch/page-open/page.content() timeout, push\_data-stall fail-loud, set\_status\_message-stall swallow) -- 137 tests green, ruff clean, pyright clean. - **Cloud QA (run `x73KS7SzTgnrM1rYs`, build 0.6.1, RESIDENTIAL proxy)**: FAILED in ~8 minutes with a loud, correctly-attributed `RuntimeError` ("exhausted independent retry budgets: bot-wall 1/4, transport 6/6") -- **not** a hang, **not** a silent zero-row success, and well short of the 30-minute deadline (the deadline governor never even had to fire; the pre-existing 0.4 retry-budget-exhaustion path did, exactly as designed). That is the correct failure mode this release exists to produce. Of the 6 transport-fault attempts, 4 were `Page.goto: Timeout 30000ms exceeded` and 1 was `NS_ERROR_PROXY_CONNECTION_REFUSED` -- proxy-exit-lottery noise, not cars.com responding -- against only 1 genuine bot-wall `challenge`. A same-day local `apify run` against the identical fixture (also RESIDENTIAL) succeeded twice with 5 real rows each in ~20-30s, and a plain-curl probe through the same RESIDENTIAL group got a fast, real HTTP 403 in 1.6s (proxy path itself is healthy, not silently falling back or timing out) -- expected, since plain HTTP has never cleared this target (CLOUD-RECON-RESULT.md); it rules out "the proxy tier is unreachable" as an explanation. Read together, this one run's failure is exit-lottery variance on a target with an independently-measured \~50% per-attempt block rate, not a new or harder block, and not a regression from this change. - **Honesty note**: this fixes the *mechanism* that turns a flaky-but- bounded target into a customer-visible hang or a silent block; it does not change cars.com's underlying block rate, which recon has previously confirmed is real and non-trivial and which this QA run reconfirmed. A run that clears enough attempts will SUCCEED with real rows (proven locally, twice); a run that exhausts its retry budgets before clearing will now FAIL LOUD with a specific, attributed reason instead of a billed, silent TIMED-OUT -- that is the fix, not a claim that every run now succeeds. A 30-minute-budget run that clears SOME pages but then goes flaky will finish SUCCEEDED with a small, honestly-labeled partial row count instead of TIMED-OUT with zero. ### 0.4 — Independent bot-wall / transport retry budgets (60%-class fix) - **Root cause, classified precisely from logs**: build 0.3.1 measured 1 SUCCEEDED / 2 FAILED (33%) across three same-session cloud runs, all billing-confirmed RESIDENTIAL. Run `LKU7Kn0xD7QDx6mUV` was a genuine, irreducible-at-that-moment bot-wall hit (3/3 attempts classified `challenge` -- cars.com actually served the interstitial each time). Run `oAXHIHssdnpdUndVH` was the real defect: 1 `challenge` (genuine) + 2 `browser-error` (`Page.goto: Timeout 30000ms exceeded` -- Camoufox never got a response from cars.com at all, a proxy-exit/transport fault) shared ONE budget of `MAX_ATTEMPTS=3`, so the run was reported "blocked" after only ONE real bot-wall attempt -- two proxy-lottery timeouts ate the other two slots. This is the exact defect class that took `manta-business-directory-scraper` from 60% to 5/5 (commit `b33368d5`, `actors/manta-business-directory-scraper/src/main.py`). - **Fixed**: `browser_fetch.BrowserFetcher` now tracks bot-wall responses (`challenge`/`geo-splash`/non-200 status -- the target genuinely answered) and transport faults (`RECOVERABLE_BROWSER_ERRORS` -- no response at all) against INDEPENDENT counters (`BOT_WALL_MAX_ATTEMPTS=4`, `TRANSPORT_FAULT_MAX_ATTEMPTS=6`, mirroring manta's measured starting point). A flaky proxy exit can no longer spend the genuine-bot-wall retry allowance, and `BrowserFetchError`'s message now names exactly which budget exhausted (plus the other class's count, for honesty) instead of a single opaque "3 attempts" figure. Backoff sleep between attempts is dropped (matching manta's proven no-sleep design) -- a fresh browser launch + warm-up already takes 10-20s, which is cool-down enough, and the removed sleep gives more attempts room inside the same run timeout. - 2 new regression tests reproduce `oAXHIHssdnpdUndVH`'s exact sequence (1 challenge + 2 transport faults must NOT exhaust the bot-wall budget) and assert the exhaustion message names the right class. 128 tests green, ruff clean, pyright clean. - **Honesty note**: `BOT_WALL_MAX_ATTEMPTS`/`TRANSPORT_FAULT_MAX_ATTEMPTS` are a reasonable starting point borrowed from manta's measured values, not yet independently measured against cars.com-specific field data -- see the post-push success-rate measurement in `docs/specs/cars-com-listings-scraper/CLOUD-RECON-RESULT.md` for whether they hold up. ### 0.3 — VDP enrichment circuit breaker (fixes a customer-visible TIMED-OUT) - **Root cause**: cloud run `KosOo4HWxHZyeC4LD` (2026-08-31, build 0.2.2, `fetchDetails=true`, `maxResults=10`) hit the platform's 600s hard timeout. Billing (`scripts/os/run_usage.py`) confirms the run genuinely used Apify RESIDENTIAL proxy (`PROXY_RESIDENTIAL_TRANSFER_GBYTES > 0`) -- the failure is NOT a proxy-tier regression. The real cause is two compounding bugs on the VDP (vehicle-detail-page) enrichment path, which was always "documented, not live-confirmed" (`CLOUD-RECON-RESULT.md`): 1. `_enrich`'s VDP fetch never passed `content_ready_selector`, so it relied on the fixed `WARMUP_SETTLE_MS` delay alone -- the same insufficient-delay bug already fixed for search pages in `0.2` (commit `251c45be`, QA run `LPrRu9dbFIER9ccrM`). Cloudflare's managed-challenge interstitial can still be on-page at capture time, misread as a real block on every single VDP fetch. 2. There was no circuit breaker: a systematically failing VDP path retried 3 Camoufox launches (with backoff) **per row**, for every row requested, with zero forward-progress signal -- turning a broken opt-in feature into an unbounded retry storm that exhausts the run's entire timeout budget instead of failing fast. - **Fixed**: `browser_fetch`'s VDP fetch now waits (bounded, best-effort) for `vdp_parser.VDP_READY_SELECTOR` (the JSON-LD script tag `parse_vdp` itself extracts from) before capturing, mirroring the search-page fix. `scraper._enrich` now trips a per-run circuit breaker (`MAX_CONSECUTIVE_ENRICH_FAILURES = 2`) that disables further VDP enrichment for the rest of the run after two consecutive failures -- remaining rows ship un-enriched (core fields intact, REQ-7) instead of retry-storming. Any enrichment success resets the counter, so an occasional real block doesn't permanently disable enrichment. - **Honesty note**: this bounds the *cost* of a broken VDP path and fixes the one confirmed timing bug in it; it does not independently re-confirm that VDP field extraction (`vdp_parser.py`'s CSS selectors) matches cars.com's live markup -- that remains documented-not- live-confirmed. The proven, revenue-bearing path (search-page scraping, `fetchDetails=false`, the QA fixture's default) is untouched by this change. - 2 new regression tests (`test_scraper.py`) reproducing the retry-storm and the reset-on-success behaviour -- 125 tests green, ruff clean, pyright clean. ### 0.2 — Camoufox engine swap + live wire-format fix (was shelved as NO-GO) - **Root cause**: this Actor's prior NO-GO ("403 on all four proxy tiers") never tried a real browser engine -- curl-cffi is a confirmed dead transport against cars.com's Cloudflare bot management on every tier (local egress, Apify datacenter, WebShare rotating residential, Apify RESIDENTIAL pinned `country_code="US"` -- `CLOUD-RECON-RESULT.md`). Camoufox + Apify RESIDENTIAL clears it (cloud run `W6l9ahDhSMCyq08lg`, 2026-08-31): real, zip-specific, server-rendered content, no interstitial. - **Also discovered during the probe**: cars.com's "srp2025" front-end redesign replaced the `div.vehicle-card` + schema.org JSON-LD shape this Actor originally hypothesized (never live-confirmed) with a `` custom element carrying a complete per-listing JSON blob -- richer and more stable than CSS-selector scraping. `parser.py`'s search-page extraction is rewritten around it; VDP (detail-page) extraction is untouched and stays documented-not- live-confirmed. - **Fetch layer replaced**: `scraper.py` no longer uses curl-cffi at all for cars.com (`browser.py` + `browser_fetch.py`, structural copy of quora-questions-scraper's Camoufox glue). Unlike quora's curl-cffi-first- then-escalate design, Camoufox is the *primary* path here -- curl-cffi has zero chance of working against this specific target, so keeping it as a first attempt would only burn a guaranteed-failing retry budget on every page. - **Cost control**: `fetchDetails` now defaults to **false** (was `true`). cars.com requires a browser render for the VDP fetch too, not just search pages -- defaulting enrichment on would cost one Camoufox render PER ROW (the exact pattern that made `bayut-uae-real-estate` a loss-maker: $62/1000 rows). Core fields (make/model/year/price/mileage/ VIN/photos) are already present from the search-page JSON island with zero extra renders; `fetchDetails=true` stays available as an opt-in for users who want the enrichment-only fields at the extra cost. - **Proxy default changed**: `proxyConfiguration` now defaults to `{"apifyProxyGroups": ["RESIDENTIAL"]}` (was the bare/no-group default, which resolves to datacenter -- confirmed 403). The stale "RESIDENTIAL has 0 availableCount on FREE" comment this default carried was itself the trap described in `reference-proxy-availablecount-trap`. - False-positive challenge markers fixed: `challenge-platform` / `turnstile` substrings are NOT blocks -- cars.com's own page legitimately loads Cloudflare's bot-management JS SDK without actually challenging the request (confirmed via two recon runs, `duheY2fCYGVBJoQGr` vs `W6l9ahDhSMCyq08lg`). - 120 tests green (`test_browser.py` / `test_browser_fetch.py` new, `test_parser.py` / `test_scraper.py` / `test_main.py` / `test_models.py` rewritten for the new wire format + fetch layer), ruff clean, pyright clean. ### 0.1 — Proxy precedence fix + geo-splash guard - **Fixed**: `_resolve_proxy_url` checked the ambient `WEBSHARE_PROXY_URL` env var *before* the Actor input's `proxyConfiguration`, so the env var silently overrode any explicit input proxy config -- including cloud probes that explicitly requested Apify RESIDENTIAL. An explicit input config now always wins; `WEBSHARE_PROXY_URL` is only a fallback for runs that don't specify one. This is why the 2026-08-15 recon's "WebShare residential" tier was the only one the Actor had ever actually run on -- Apify RESIDENTIAL had never been genuinely probed (`docs/specs/cars-com-listings-scraper/CLOUD-RECON-RESULT.md`). - **Added**: geo-splash guard (`_is_off_site_redirect` in `scraper.py`) -- an HTTP 200 that lands off `www.cars.com` is now treated as a block (rotate session + retry), not parsed as data. Belt-and-suspenders for the existing `country_code="US"` pin on the Apify proxy config (`feedback-pin-proxy-country`). - 12 new regression tests (`test_main.py` proxy precedence, `test_scraper.py` geo-splash guard) -- 125 tests green, ruff clean. ### 0.0 — Initial release - Full scraping implementation: dual JSON-LD/`vehicle-card` CSS extraction for search results, per-field merge, VIN charset validation, and vehicle-detail-page (VDP) enrichment split across `parser.py` / `vdp_parser.py`. - `curl-cffi` fetch layer (`scraper.py`) with rotating browser impersonation, Cloudflare-style block/challenge detection, and exponential backoff/retry. - Per-listing and per-page fault isolation: a malformed card, a missing field, or a failed detail-page fetch degrades one row, never the run. - Pydantic v2 `ActorInput` / `ResultRow` models (`models.py`), PPE charging (`actor-start` + `result-row`), and the zero-emitted-row fail-loud backstop in `main.py`. - 113 unit/fixture tests across `test_models.py`, `test_parser.py`, `test_vdp_parser.py`, and `test_scraper.py` -- zero live network calls, synthetic fixtures only (`LOCAL-RECON-RESULT.md`).