# Changelog of coches.net Spain Car Scraper (`devilscrapes/coches-spain-cars`) Actor - **URL**: https://apify.com/devilscrapes/coches-spain-cars/changelog.md - **Full Actor documentation**: https://apify.com/devilscrapes/coches-spain-cars.md ## coches.net Spain Car Scraper — Changelog ### 0.9.0 — 2026-08-26 - **Stop failing runs for a genuine zero-match search (30-day public success rate 72.4% — 6 FAILED + 2 TIMED-OUT of 29 — after v0.8.1's cold-detail-session fix already verified live).** Checked the three sibling-actor root causes fixed this week first: sticky proxy session (already NOT present — `_new_proxy_url` mints a fresh `session_id` on every `_rotate`), unpinned proxy country (already pinned to `ES` since v0.8.0), and proxy tier (confirmed RESIDENTIAL via `run_usage.py` billing on run `iFBVbdDJS8Uap1707`, not just requested). None applied — a direct cloud repro (run `zvkIb3NZtRNnk5ZQp`, default `maxResults=50`/`enrichDetails=true`) SUCCEEDED cleanly with 50/50 rows, 49/50 enriched, confirming the existing anti-bot handling works. Root cause found by reading, not reproducing: `_iter_listings` treated **any** zero-items page-1 result as a failure — a fetch that never returned content, an unparseable/blocked page, AND a page that genuinely parsed and proved zero matches (e.g. a customer's narrow `searchUrl` filter combo, such as an obscure make/year/price range) all raised the same `EmptySearchResultError`, paying the customer's `actor-start` charge for a FAILED run when the target had actually answered correctly. This is the fleet-wide "empty is not a failure" pattern (see `ops/os/EMPTY-IS-NOT-A-FAILURE-2026-08-19.md`) applied to this Actor. - Added `parser.is_confirmed_empty_search()` — a positive-signal check requiring `initialResults` to be a dict, `items` to be present as an empty list, AND `totalResults` to be present as the integer `0`. Anything short of that exact shape (missing `initialResults`, a contradictory non-zero `totalResults` alongside empty `items`, or a parse/fetch failure) stays fail-loud — unchanged from v0.8, and still carries the raw HTML to the KVS for debugging. `_SearchPageResult` now carries `confirmed_empty`, and `_handle_empty_page_one` is the single decision point: confirmed-empty finishes the generator cleanly (SUCCEEDED, 0 rows, `Actor.set_status_message` already reports "Done — 0 car listing(s) scraped." unchanged); everything else still raises exactly as before. See `tests/test_confirmed_empty_search.py` (parser-level, 7 cases) and `tests/test_empty_search_result.py` (caller-level; the two pre-existing fail-loud tests are unchanged, plus two new cases pinning both directions of the fix). - No changes to proxy/session/retry/geo-block logic — those were already correct. - **Also fixed while gating this release**: the `actor-publisher` pre-publish checklist's charge-order gate (`scripts/verify_charge_order.py`) flagged `main.py` as charging `actor-start` before validating input — a pre-existing condition unrelated to the empty-search fix above, but this Actor is being touched/published now so the gate's ratchet applies. `scraper.run()` is an async generator; its internal `ActorInput.model_validate()` call doesn't execute until the first `__anext__()`, which happens *after* `main()` already charged. A malformed input therefore billed the customer for a run that immediately raised. `main()` now calls `ActorInput.model_validate(actor_input)` directly before `_charge("actor-start", ...)` — cheap, pure-Python, no I/O. See `tests/test_main_charge_order.py`. ### 0.8.0 — 2026-08-06 - **Fix detail pages rejecting "cold" requests with a persistent HTTP 403 (30-day public success rate still 17% — 21 FAILED/4 TIMED-OUT/5 SUCCEEDED of 30 — after v0.7's ProxyError fix).** Reproduced directly against the live Actor: cloud run `ezxrbObUgW57S5zXg` (build 0.7.1) shows the search page fetching real content cleanly, then *every* subsequent detail-page fetch getting HTTP 403 — even after the existing retry loop rotated proxy IP **and** browser-impersonation profile on each attempt. Root cause: the module's own docstring always documented the intended design ("GET the search listing first to warm the session's cookies, then issue detail requests on the same session with a Referer") but the implementation never did this — `_get_with_retry` opened a brand-new, cookie-less `AsyncSession` for *every single* HTTP request since the Actor's very first commit. coches.net's anti-bot layer reliably rejects a detail-page "click" from a session that never visited the search results first, regardless of which residential IP it comes from. - Replaced the per-request session with `_RotatingSession`: one curl\_cffi session (and its cookies) now persists across a search-page fetch and all of that page's detail-page fetches — like a real visitor browsing from results into ads — and is only replaced (and re-warmed with a fresh search-page revisit before the next request) when the target actually blocks it. See `tests/test_session_reuse.py`. - **Pin the proxy exit country to Spain by default** (`apifyProxyCountry` defaults to `"ES"` unless the customer overrides it) — coches.net is Spain-only content; an unpinned residential exit can land geo-random (see memory `feedback-pin-proxy-country`). See `tests/test_proxy_guard.py`. - **Add a geo-splash/off-site-redirect guard**: a 200 response whose final URL left `www.coches.net` (a geo-mismatched exit redirected to a country-selector/splash host) is now treated as a block — same retry-and-rotate path as a 403 or the existing bot-detection soft block — instead of being handed to the parser as real content. See `tests/test_geo_block_detection.py`. - **Never a silent empty success**: zero listings parsed off search page 1 now raises `EmptySearchResultError` carrying the raw HTML, which `main.py` dumps to the Actor's key-value store (`EMPTY_SEARCH_PAGE_HTML`) before failing the run loud — so a wrong-locale/challenge page that slips past every rotation is inspectable after the fact instead of just producing a bare "no listings found" error. See `tests/test_empty_search_result.py` and `tests/test_empty_search_debug_dump.py`. ### 0.7.0 — 2026-08-05 - **Fix the ProxyError-misclassified-as-permanent bug (30-day public success rate 13%, 21 FAILED + 5 TIMED-OUT of 30 — still failing after v0.6's soft-block fix).** Root cause: v0.5 (see below) correctly identified that a genuine proxy-*authorization* rejection is a permanent, whole-run condition and should fail loud instead of retrying — but its fix caught **every** `curl_cffi` `ProxyError`, not just that specific 407-class failure. In production, the dominant `ProxyError` signature is `curl: (56) CONNECT tunnel failed, response 590` — a **per-session** tunnel failure against this specific target/exit IP (libcurl error 56, a generic network-receive error), not an account/plan authorization problem. `_try_get` raised `PermanentProxyError` on the first occurrence of *any* `ProxyError`, killing the whole run instantly — including mid-run, after the retry loop had already successfully rotated past two real HTTP 403s on the same detail page. Reproduced directly against the live Actor: 3/3 cloud runs with the default input (`maxResults: 50, enrichDetails: true`) FAILED in ~18-20s with this exact signature; one additional run recovered from two 403s via the existing retry/rotate path and then died on the third attempt's `ProxyError`. Confirmed `curl_cffi.curl.CurlECode` 56 = `RECV_ERROR`, a generic network condition, not a distinct auth-failure code — genuine proxy-auth rejections surface with `407` in the message instead. - `_try_get` now only raises `PermanentProxyError` when the `ProxyError` message carries the `407` auth-failure marker (`PROXY_AUTH_FAILURE_MARKER`). Every other `ProxyError` maps to `HTTP_UNAVAILABLE` and flows through the existing retry-and-rotate path — a fresh session on the next attempt gets a different residential IP, which the direct repro confirms frequently succeeds (search-page fetches with real residential IPs returned genuine `__INITIAL_PROPS__` content with zero retries in isolated small runs). - Added a run-wide wall-clock budget (`MAX_RUN_SECONDS = 600`) via a new `_FetchContext` dataclass threaded through `_iter_listings`/`_emit_page`. Now that more requests retry instead of instantly killing the run, this bounds worst-case duration so a heavily-blocked run fails loud (zero results) or returns a partial dataset well before an external platform/customer timeout would silently kill it as TIMED-OUT — directly addressing the 5/30 TIMED-OUT runs alongside the 21 FAILED. - Added regression tests: `test_proxy_error_fail_fast.py` now covers the exact "response 590" production signature (must retry, not raise) and a 407-marked variant (must still raise `PermanentProxyError`). `test_run_budget.py` covers the new deadline behavior in both `_iter_listings` (fail loud on zero results) and `_emit_page` (stop cleanly with partial results). - No changes to the v0.6 soft-block detection — this is additive. ### 0.6.0 — 2026-07-30 - **Fix the HTTP-200 bot-detection soft block (30-day public success rate 10%, 27/30 FAILED/TIMED-OUT — unchanged from the day before at 28/30).** Root cause: coches.net's bot-detection layer sometimes answers a blocked request with HTTP 200 and a small "Ups! Parece que algo no va bien..." interruption page instead of a 403/429. `_try_get` only branched on `response.status_code`, so a 200 was always treated as a successful fetch — the challenge page's body was handed to `parse_initial_props`, which found no `__INITIAL_PROPS__` and returned `None`. On page 1 that raised a misleading "no listings found" `ValueError` (a FAILED run); on later pages it silently ended iteration early (a truncated but "successful" run, or a TIMED-OUT run once retries piled up across many detail-page enrichments). Neither path ever rotated the session or retried, even though the existing 403/429 retry-and-rotate machinery would have handled it identically if only it had been told about it. Reproduced directly against the live site (repeated requests from the same network eventually get served this exact interruption page). - Added `parser.is_bot_block_page()` — detects the interruption page via three stable markers in its markup (`showBlockPage`, `onProtectionInitialized`, `interruption-message`). `_try_get` now checks this on every 200 response and returns the new `HTTP_SOFT_BLOCK` sentinel instead of the body when detected, routing it through the same retry/rotate path as a real 403/429. - Added `tests/test_bot_block_detection.py` (captured interruption-page fixture + `_try_get` unit coverage) pinning the regression. - No proxy-guard or proxy-error-handling changes in this release — this is additive to the v0.4 RESIDENTIAL guard and v0.5 `PermanentProxyError` fail-fast, both already live. ### 0.5.0 — 2026-07-10 - **Fix TIMED-OUT customer runs (30-day public success rate 15%, 28/33 TIMED-OUT).** Root cause: when a run's Apify Proxy configuration requests a group the customer's plan can't authorize (RESIDENTIAL with 0 GB quota is common on FREE-tier accounts), Apify Proxy answers with a permanent proxy-auth rejection for the *entire run*. `curl-cffi` surfaces this as `ProxyError` (libcurl `CURLE_PROXY` — HTTP 407-class), which the old code caught with a broad `except (OSError, RuntimeError)` and mapped to HTTP 503 — a *transient*, retryable status. Every single search/detail request then burned the full 5-attempt exponential backoff (up to ~60s of sleeps, compounding across up to 100 pages and 1000 detail-enrichment requests), blowing past the Actor's default run timeout without ever raising — hence TIMED-OUT, not FAILED, and zero visibility in our own run history (the failures all belonged to customers). - Added `PermanentProxyError` in `scraper.py`. `_try_get` now catches `curl_cffi.requests.exceptions.ProxyError` specifically and raises it immediately instead of mapping it to a retryable status — the Actor now fails loud on the first proxy-auth rejection instead of silently exhausting the retry budget on a condition that cannot self-heal within the run. - Added `tests/test_proxy_error_fail_fast.py` — pins the exact regression (ProxyError -> fail fast) and confirms genuine transient network errors keep their existing retry semantics. - Added `tests/fixtures/input.qa.json` (small, `enrichDetails: false`) for cloud QA. ### 0.4.0 — 2026-06-10 - Force-residential proxy guard: `_resolve_proxy_configuration` now always injects `RESIDENTIAL` into `apifyProxyGroups` when the group is missing, empty, or set to a datacenter-only group — preventing the HTTP 403 that customers hit when running without residential proxy. - Fail-fast on proxy unavailable: if `Actor.create_proxy_configuration` raises or returns `None` the Actor exits immediately with a clear `RuntimeError` describing the required plan/config, rather than silently returning empty results from datacenter exits. - `country_code` (`apifyProxyCountry`) preserved through the guard. - Updated `models.py` proxy field description and `input_schema.json` proxy description to document the RESIDENTIAL requirement. - Added `tests/test_proxy_guard.py` with unit coverage for all guard branches. ### 0.3.0 — 2026-06-05 - Config-refresh re-push: RESIDENTIAL proxy group explicitly set in both `input_schema.json` `default` and `prefill` so Apify auto-QA receives residential exits rather than datacenter IPs, clearing the 403 / "Under maintenance" trap triggered by DataDome-class anti-bot on coches.net. - No code changes — build is functionally identical to 0.2; only the published proxy config is updated. ### 0.2.0 — 2026-06-03 - Per-attempt proxy session rotation: each retry now opens a fresh `AsyncSession` with a new `session_id` (new residential exit IP) and a randomly-picked browser impersonation profile, defeating persistent IP/fingerprint blocks. - Removed the single static proxy URL resolved at startup; the `ProxyConfiguration` object is now threaded through to the scraper for per-request fresh URLs. - Fixed `Dockerfile` `CMD` to use `python -m src` (via `__main__.py`). - Fixed `output_schema.json` templates — removed forbidden `clean=true` param. - Added `.actorignore` to exclude `tests/`, `scripts/`, `storage/`, `.venv/`. ### 0.1.0 — 2026-05-15 - Initial scaffolded release.