# Changelog of Vinted Listings & Condition Scraper (`devilscrapes/vinted-listings-scraper`) Actor - **URL**: https://apify.com/devilscrapes/vinted-listings-scraper/changelog.md - **Full Actor documentation**: https://apify.com/devilscrapes/vinted-listings-scraper.md ## Vinted Listings & Condition Scraper — Changelog ### 0.3 — 2026-09-30 (fix on branch, NOT yet pushed — see notes below) - **Root cause confirmed and fixed: Vinted retired `/api/v2/catalog/items`.** The 30-day daily-report flag and Apify QA run `sMC16aDTvIygTTedw` (build 0.1.3, exit 91) both point at `GET /api/v2/catalog/items -> HTTP 404`. Direct curl-cffi probing (2026-09-30, no Apify Proxy — account is invoice-blocked, proxy is down — probed straight from this machine) confirms this is a retired route, not an anti-bot block: `/api/v2/countries` and `/api/v2/users/1` on the *same* session both still return 200, the 404 body is Vinted's own branded not-found template with real origin headers (`x-envoy-upstream-service-time`, `x-runtime`) — i.e. the request reaches their backend and is cleanly rejected, not stopped at an edge/WAF — and the literal string `/catalog/items` no longer appears anywhere in Vinted's shipped frontend JS. This corrects and supersedes the 0.2 changelog entry's tentative "Cloudflare gate" theory, which that entry's own same-day correction had already walked back to "retired route" without a fix in hand. - **The replacement is Vinted's server-rendered `/catalog` page**, not a new JSON endpoint — there isn't one. `GET /catalog?search_text=...` (plus `page`/`price_from`/`price_to`, still genuine functioning query params — verified: identical `search_text` request on `page=1` vs `page=2` returns disjoint item sets, and `price_from=50&price_to=70` genuinely filters server-side) returns a Next.js page whose React Flight payload embeds the full catalog item array as real, structured JSON inside a `self.__next_f.push([1, "..."])` script call. `per_page` is accepted but ignored — the server always returns 96 items/page, which happily already matches `parser.PER_PAGE_MAX`. - `src/client.py`: `CATALOG_PATH` now points at `/catalog`; `fetch_catalog_page` parses the HTML response instead of calling `.json()`. New `_extract_catalog_payload`/`_iter_flight_chunks`/ `_find_items_array`/`_scan_balanced_json` do the extraction (pure, regex + balanced-bracket scan — no HTML parser dependency needed). `DEFAULT_HEADERS`'s `Accept` header changed from `application/json` to a real-browser `text/html,...` value to match what's actually being requested now. - `src/client.py`: `_handle_error_status` now distinguishes the two non-2xx cases the task called out. **HTTP 404** raises the new `VintedRouteRetiredError(VintedRequestError)` immediately, no retry — the route is gone, not blocked, and retrying it burns attempts for nothing. **HTTP 403** moved from "raise immediately, no retry" (the v0.1-era assumption, encoded in the now-replaced `test_fetch_catalog_page_403_raises_immediately_no_retry`) into `ROTATE_STATUSES` alongside 408/429/503 — a 403 is a challenge, so it now rotates browser-impersonation profile + proxy session and retries, same as the other retriable statuses. - `src/parser.py`: rewritten against the new SSR item shape (`{"id", "productItem": {"title", "price", "photos", "itemBox": {"accessibilityLabel": "...Brand: X, Condition: Y, Size: Z..."}}}`). `visible_items` is now a thin pass-through — the SSR page never serializes a hidden item into the array at all, so REQ-6's filter is enforced server-side and there is nothing left to check locally. Brand/condition/size are parsed out of `itemBox.accessibilityLabel` (the only structured source left; `itemBox.firstLine`/`secondLine` looked like brand/size candidates but fall back to the item title when no real brand is set, so they're not a reliable signal — verified against a real captured item that has no brand). **Capability lost in the migration, called out rather than silently dropped:** `seller_login` is always `None` now — the SSR catalog card exposes only `user.id`, no username, and there is no other public source for it in v1. - `src/scraper.py`: `build_row` now needs the run's `domain` (to turn `productItem.url`'s domain-relative path into an absolute URL), so `_rows_from_items` takes `domain` and passes it through. - Tests: `tests/test_client.py`, `tests/test_parser.py`, `tests/test_scraper.py` rewritten against the new shape. New regression tests pin (a) `CATALOG_PATH == "/catalog"` and `!= "/api/v2/catalog/items"` (the route itself), and (b) the 404-vs-403 branch (`test_fetch_catalog_page_404_raises_route_retired_immediately_no_retry` / `test_fetch_catalog_page_403_rotates_profile_and_retries`). Fixtures `catalog_success.html`/`catalog_empty.html` are built mechanically from a real captured response (breadth-trimmed to 3 real items, not invented — see `tests/test_client.py`'s fixture-provenance docstring); `catalog_items_404.html` is the real, byte-for-byte 404 body. The stale hand-authored `catalog_items_page1.json` / `catalog_items_mixed_visibility.json` / `catalog_items_empty.json` (old flat JSON-API shape — see `reference-hand-authored-fixtures-encode-fiction`) are removed. 87 local tests pass (ruff + pyright clean); `-m smoke` opt-in test hits the real, current `/catalog` route end-to-end. - **`actors/vinted-sold-listings` needs the identical fix.** It imports a byte-for-byte copy of this same `src/client.py` (confirmed via `diff`, still identical pre-fix) and fails the exact same way on the exact same retired endpoint. Out of scope for this session (task explicitly scoped to `vinted-listings-scraper`) — flagging for a follow-up fixer invocation on that slug; the fix here should port directly since the response shape is per-domain, not per-Actor. - **Not pushed.** The Apify platform remains invoice-blocked (403 "too many outstanding invoices") — `apify push`, cloud builds, cloud QA, and Apify Proxy are all refused/down. Verified entirely with direct local probing + the local test suite. Do not stamp this as published until the block lifts and both gates (11-gate preflight + cloud QA) run clean. ### 0.2 — 2026-09-28 (fix on branch, NOT yet pushed — see notes below) - Fix (fault isolation): `src/scraper.py` `run()` now catches `VintedRequestError` per keyword instead of letting it propagate out of the generator. Root cause of the daily-report flag ("53% success, 13/28 customer runs FAILED in 30d"): when a run has more than one `keywords` entry, a single keyword whose catalog fetch permanently fails (retry-exhausted `VintedRequestError` — `client.py` retries only `408/429/503`/network errors, not other 4xx/5xx) raised uncaught out of `scraper.run()`, aborting the whole generator and crashing the entire run — discarding rows already scraped for every *other* keyword in the same run and marking a customer run `FAILED` even when it had already produced good data. Reproduced locally with a new failing test (`test_one_keyword_exhausting_retries_does_not_abort_remaining_keywords`) before the fix; now green. This matches the fleet-wide "one bad item kills the whole run" pattern (`reference-fleet-fault-isolation-pattern`). A run where *every* keyword fails this way still correctly fails loud via the existing REQ-8 zero-rows path in `main._finish` — only genuinely partial failures now survive with whatever data landed. - **Not pushed today.** The Apify platform is returning `403 platform-feature-disabled` ("too many outstanding invoices") on `apify push` / builds / runs — day 3 of a billing block only the CEO can clear. This fix is committed to `fix/vinted-listings-scraper-fault-isolation` and verified with the full local suite (ruff, pyright, pytest — 76 passed) but has **not** been pushed or cloud-QA'd. Do not stamp this as published until both gates run clean once the block lifts. - **Separate, likely more severe finding — NOT fixed, needs follow-up with a real browser trace.** Local probing (no cloud run possible) shows `GET https://www.{domain}/api/v2/catalog/items` now returns a byte-identical HTTP 404 (mislabeled `content-type: application/json`; real body is Vinted's themed "not found" page, exactly 3771 bytes) across every combination tried: 4 curl-cffi browser-impersonation profiles (`chrome131`, `chrome124`, `firefox147`, `safari180`) x 2 proxy pools (direct + our WebShare proxy) x several header variants (`Referer`, `X-Requested-With`, `Sec-Fetch-*`, `X-Money-Object`). The cookie-seed GET still succeeds (200, real page, sets `access_token_web`/`refresh_token_web`) and sets Cloudflare's `__cf_bm` bot-management cookie but never `cf_clearance` — consistent with the `project-camoufox-track` memory pattern (Cloudflare Bot Management gates a route behind a JS challenge that curl-cffi cannot solve regardless of TLS fingerprint). This is a change since 2026-08-01, when the same endpoint/stack was cloud-QA-verified `SUCCEEDED` with 5/5 real rows (see this file's `0.1` entry and `docs/specs/vinted-listings-scraper/notes.md`). **Not fixed** — I could not test Apify's actual RESIDENTIAL proxy pool (billing block prevents any cloud run) so I cannot rule out that Apify's residential exits still clear this check, and migrating to a JS-capable fetch path (Camoufox) is a real architecture change (new dependency, materially higher compute cost) that needs its own decision, not a same-day minimal fix. Flagging with medium-high confidence as the probable dominant contributor to the reported failure rate; see this session's report for full probe evidence. - **CORRECTION, same day (2026-09-28): the Cloudflare theory above is DISPROVEN.** The sibling investigation on `vinted-sold-listings`, which hits a byte-identical `client.py`, ran the control this entry was missing: on the *same* session that 404s, `GET /api/v2/countries` and `GET /api/v2/users/1` both return **200 — and neither has `cf_clearance` either.** So a missing `cf_clearance` cannot be what distinguishes the failing route. The 404 also carries **origin** headers (`x-envoy-upstream-service-time`, `x-runtime`) and Vinted's own branded 404 template with correct hreflang alternates for all four TLDs — the request is reaching their Rails/Envoy backend, not being stopped at Cloudflare's edge. Finally, the literal string `/catalog/items` no longer appears anywhere in Vinted's shipped frontend JS; it is replaced by `/catalog/blocks` (feature-flagged) or `ApiGatewayClient.get("/svc-catalogue/items")`. **Vinted retired the endpoint. It is a dead route, not a bot gate.** Do NOT fund a Camoufox migration for this: a real browser cannot reach a route that no longer exists. The fix is to capture the current request from a JS-executing browser session and repoint the shared `client.py` for both Vinted Actors. Recorded here because this entry, left alone, pointed the next session at an expensive architecture change that could not have worked. ### 0.1 — 2026-08-01 - Fix: default proxy configuration now requests the `RESIDENTIAL` Apify Proxy group (`.actor/input_schema.json` `default`/`prefill`, `src/models.py` `ActorInput.proxy_configuration` default). Root cause of the reported failures: Vinted's cookie-seed GET (`GET https://www.{domain}/`) returns an instant, non-retriable `HTTP 403` on datacenter/auto Apify Proxy exits — confirmed via a cloud reproduction run (default proxy config: `FAILED`, same input with `apifyProxyGroups: ["RESIDENTIAL"]`: `SUCCEEDED`). A caller- supplied `proxyConfiguration` still overrides this default — REQ-2's "SHALL NOT hard-require RESIDENTIAL" still holds; only the *default* changed, matching `DEVILSCRAPES-CONTEXT.md`'s "Anti-blocking stack" §2 rule for anti-bot-prone targets. - `tests/fixtures/input.qa.json` updated to request `RESIDENTIAL` explicitly, matching the new default and the verified-working cloud repro input. - `tests/test_models.py`: replaced `test_default_proxy_never_forces_residential` (encoded the buggy behavior) with `test_default_proxy_defaults_to_residential` + `test_explicit_proxy_configuration_overrides_default`. ### 0.0.1 — 2026-07-29 - Scaffolded: `.actor/` (`actor.json` with `categories: ["ECOMMERCE"]`, `input_schema.json` / `dataset_schema.json` / `output_schema.json` matching design.md field-for-field, `Dockerfile`), `src/main.py` placeholder (reads input, validates against the placeholder `ActorInput`, pushes one clearly-marked placeholder record, exits clean), `src/__init__.py` / `src/__main__.py`, test scaffold (`tests/conftest.py`, `tests/test_main.py`), the QA fixture (`tests/fixtures/input.qa.json`, spec's own live-verified `keywords=["jacket"]`/`domain="vinted.com"`/`maxItems=5` combination), README skeleton, `scaffold.json`, `promo.json`. - `.actor/icon.svg` deferred to `actor-icon-designer` (Vinted brand mark inside the canonical Devil Scrapes container). - `.actor/pay_per_event.json` deferred to `implementer` (T07) per `docs/specs/vinted-sold-listings/design.md`'s file plan. - Added throwaway placeholder `src/models.py` (bare `ActorInput`/ `ResultRow`, no validators) and `src/scraper.py` (`NotImplementedError` stub) so the repo's ADR-0004 Pydantic-wiring gate (`scripts/verify_pydantic.py`, enforced by `.claude/hooks/stop_check.sh`) passes at scaffold time. Both are fully replaced, not extended, by T02 (`models.py` — the real validated `ActorInput`/`ResultRow`) and T04-T06 (`parser.py`/`client.py`/`scraper.py` — the real orchestration) per `docs/specs/vinted-sold-listings/design.md` and `tasks.md`. - Real crawler logic (`src/client.py`, `src/parser.py`, plus the real `src/models.py` / `src/scraper.py` / `src/main.py`) is not implemented yet — see `docs/specs/vinted-sold-listings/tasks.md`.