# Changelog of ZipRecruiter Jobs Scraper (`devilscrapes/ziprecruiter-jobs-scraper`) Actor - **URL**: https://apify.com/devilscrapes/ziprecruiter-jobs-scraper/changelog.md - **Full Actor documentation**: https://apify.com/devilscrapes/ziprecruiter-jobs-scraper.md ## ZipRecruiter Jobs Scraper — Changelog ### 0.3 — 2026-09-09 #### Fixed — 30-day customer success rate 22% (7/9 FAILED), all "blocked by target anti-bot" CEO daily report flagged this listing (`publicActorRunStats30Days`: TOTAL 9, SUCCEEDED 2, FAILED 7). Per this fleet's "check the current build first" protocol, reproduced live on the currently-deployed build (`0.2.1`, unchanged since 2026-08-31) before touching any code: run `8D8FTlOX9ZQOdw69f` (fixture: `searchQuery="software engineer", location="Austin, TX"`) FAILED with `blocked by target anti-bot — 0 page(s) fetched` after exhausting all 4 retry attempts, each logging `blocked (Cloudflare challenge) — rotating session`. Confirms the failure survives the current build — not residue from an older one. **Root cause**: `browser.py`'s `launch_browser()` never passed `geoip=True` to `AsyncCamoufox`, per an org-wide lesson diagnosed on `ai-overview-citations` (v0.9, FREE-tier proxy pool): `geoip=True` sweeps third-party IP-echo services through the proxy pre-launch, and that sweep crashed the run often enough on the FREE-tier shared pool to be worth avoiding outright. This account has been on STARTER since 2026-08-20, and this Actor's proxy is a real Apify RESIDENTIAL exit (confirmed via `run_usage.py`-style billing on the FAILED run: `PROXY_RESIDENTIAL_TRANSFER_GBYTES = 0.0157`) — so without `geoip=True`, Camoufox's browser fingerprint (locale/timezone) never matched the real geolocation of that residential exit, which is itself a bot signal Cloudflare can score on. A 2026-09-09 local recon (5 fresh Camoufox launches against the real target through Apify RESIDENTIAL, `US`-pinned, run from this machine — STARTER unlocked local proxy access) found the Cloudflare Managed Challenge clearing 4/5 times with `geoip=True` vs. 0/1 without it, with **zero** `InvalidIP`/`InvalidProxy`/`LocaleError` launch crashes across all 5 launches — the FREE-tier crash lesson does not reproduce on this account's tier (matching cross-Actor evidence already recorded for `aliexpress-products-scraper`/`glassdoor-reviews-scraper`). **The fix**: `launch_browser()` now passes `geoip=True` whenever a proxy is attached (never without one — there is no exit geography to self-detect). Even in the worst case where a launch does crash, `InvalidIP`/`InvalidProxy`/ `LocaleError` were already in `RECOVERABLE_BROWSER_ERRORS`, so `client.py`'s existing per-attempt retry/rotate loop absorbs it as one lost attempt, not a failed run. Two new regression tests (`test_launch_browser_passes_geoip_when_proxy_attached`, `test_launch_browser_omits_geoip_and_proxy_without_a_proxy`) lock in the proxy-gated behavior via a mocked `AsyncCamoufox`, matching the pattern established in `yelp-business-reviews-scraper`/`glassdoor-reviews-scraper`. 143 tests pass (was 140), ruff clean, pyright clean on every file this change touches (`src/browser.py`, `tests/test_browser.py`) — the two pre-existing `recon_cloud.py` pyright errors (missing `curl_cffi` import, a leftover diagnostic script from before the Camoufox migration, not the shipped entrypoint) are unrelated drift, present before this change too. ### 0.2 — 2026-08-31 #### Verified - Pushed build `0.2.1`. Cloud QA (`scripts/os/qa_sweep.py`): run `m8ULAINxsI06Kxgo1` SUCCEEDED, 3 rows, PPE charged `{"actor-start": 1, "result": 3}`. `run_usage.py` confirms `PROXY_RESIDENTIAL_TRANSFER_GBYTES > 0` — RESIDENTIAL actually used. Dataset items eyeballed directly: all 3 rows carry real company/location/structured salary/posted\_at and a canonical `/c//Job//-in-?jid=...` `source_url` — none of the `878bvyzeAjSPJXY0A` garbage shape. Stays PRIVATE — publish/isPublic is a separate human decision. #### Fixed — the actor delivered garbage rows while reporting SUCCEEDED Run `878bvyzeAjSPJXY0A` (2026-08-30, build 0.1.1) SUCCEEDED with 3 rows that were not jobs: a title "ZipRecruiterZipRecruiter" linking to `/`, a company's generic `/Jobs/` listing page mislabeled with the company name as the job title, and the search page's own URL echoed back. The immediately-following run `WWpsMfqQfdOf7B7Dv` SUCCEEDED with 0 rows. Root cause, confirmed via a fresh local recon (Camoufox + RESIDENTIAL, 2026-08-31): - **curl-cffi cannot clear ZipRecruiter's Cloudflare Managed Challenge — ever, on this target.** 0/10 across all 4 `BROWSER_PROFILES`, on RESIDENTIAL, in a fresh recon — matching the 2026-08-06 cloud recon (`docs/specs/ziprecruiter-jobs-scraper/CLOUD-RECON-RESULT.md`) on both proxy tiers. This is a genuine JS-execution wall (Cloudflare's challenge page requires a real browser to resolve a client-side JS proof-of-work), not an HTTP-fingerprint or proxy-tier problem — the 0.1 fix's move to RESIDENTIAL could never have worked on its own. `878bvyzeAjSPJXY0A`'s 3 "rows" came from `html_fallback.py`'s generic `CARD_SELECTORS` (including a bare `article`) latching onto page furniture in whatever non-challenge, non-results 200 response occasionally slips through — `JOB_URL_MARKER` (added 2026-08-30, commit `6125bb34`) correctly turns that class of response into a zero-postings page instead of a garbage one, but it never fixed extraction against a REAL results page, because neither `jsonld.py` nor `html_fallback.py` had ever seen one. - **The parser's fixtures were synthetic and wrong.** `jsonld.py` assumed ZipRecruiter's `ItemList` nests each node under `itemListElement[].item` (schema.org's documented shape). The real page's `ItemList` is FLAT — `{"@type": "ListItem", "name": ..., "url": ...}`, no `item` wrapper, no `JobPosting` sub-object — so `jsonld.py` silently extracted zero postings from every real page and fell through to `html_fallback.py` every time. **The fix:** switched the transport from curl-cffi to Camoufox (ADR-0002's browser layer) — the only thing capable of clearing this specific wall — and added `src/parser/next_flight.py`, which parses ZipRecruiter's REAL wire format: once Camoufox clears the challenge, the server-rendered page embeds every job as a JSON object inside a Next.js RSC `self.__next_f.push(...)` flight payload (title, company, location, structured salary, employment type, remote/in-person flag, posted date, and the confirmed-live canonical `/c//Job//-in-?jid=...` URL — all in one place, no text re-parsing needed). This is now the PRIMARY extraction path; `jsonld.py` (fixed to also handle the real flat `ListItem` shape) and `html_fallback.py` stay as fallbacks. Also fixed two URL-building bugs the same recon caught: the search radius param is `radius`, not the invented `radius_miles`; and pagination is a path segment (`/jobs-search/`), not a `page=` query param. `src/browser.py` (new) owns the Camoufox/Playwright primitives, including a poll loop that waits out the Cloudflare challenge's 6-18s clear time (a plain `page.goto()` returns before the challenge resolves). Bumped `minMemoryMbytes`/`defaultMemoryMbytes` (256→2048) and switched the Dockerfile base image to `apify/actor-python-playwright:3.11` — Camoufox needs the Playwright base image's system libraries and materially more memory than the curl-cffi-only build. ### 0.1 — 2026-08-27 #### Fixed - **Default proxy resolution never used RESIDENTIAL.** `_build_proxy_factory()` passed `groups=spec.get("apifyProxyGroups") or None` — with no explicit override (the QA fixture and every real customer input we've seen), `None` resolves to Apify's automatic/datacenter pool. `run_usage.py` on both the last SUCCEEDED (`kLwRkRaHHl4IeWAWZ`) and the immediately-following FAILED (`QmcKcOo9IjmrvLKXW`) run confirms zero `PROXY_RESIDENTIAL_TRANSFER_GBYTES` on either — neither run touched the residential tier, despite it being entitled on this account since the 2026-08-20 STARTER upgrade. ZipRecruiter's Cloudflare front clears far more reliably on RESIDENTIAL exits than on the automatic pool, matching the escalation ladder that already worked for idealista/amazon-reviews/bayut. Now defaults to `apifyProxyGroups: ["RESIDENTIAL"]` + `apifyProxyCountry: "US"` (ZipRecruiter is a US board; a geo-random exit risks a silent wrong-country 200, not an honest error) unless the caller explicitly overrides. `input_schema.json`'s `proxyConfiguration` default/prefill updated to match so the Store UI is honest about what a customer gets by default. - Fixed a stale assertion in `test_pay_per_event_matches_spec_exactly` — the 2026-08-26 pricing commit (`0463e80`) live-priced the Actor at `actor-start` $0.20 / `result` $0.0015 but left the test asserting the old $0.03 / $0.0012 placeholder values, which is a local-red regression unrelated to this fix but blocking a clean gate run. - Ruff `UP017` autofix in `tests/test_scraper.py` (pre-existing, unrelated). #### Known limitation - Not yet cloud-verified. The account's monthly usage hard cap was exceeded ($29.58 / $29, cycle 2026-08-19 → 2026-09-18) at investigation time — every `apify api` run-start attempt returns `platform-feature-disabled: Monthly usage hard limit exceeded`, including on the already-pushed build. This fix cannot be cloud-QA'd or pushed until the cap clears or is raised. ### 0.0.1 — 2026-08-06 - Scaffolded skeleton (T01). Boots, reads input, pushes one placeholder dataset row. Real curl-cffi/parser pipeline pending (T02-T11 per docs/specs/ziprecruiter-jobs-scraper/tasks.md).