# Changelog of Upwork Job Leads Scraper (`devilscrapes/upwork-job-leads-scraper`) Actor

- **URL**: https://apify.com/devilscrapes/upwork-job-leads-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/devilscrapes/upwork-job-leads-scraper.md

## Upwork Job Leads Scraper — Changelog

### 0.6 — 2026-09-18

- **Fix (billing correctness): duplicate `job_id`s were double-billed
  within a single run.** Cloud run `CK7RlxycwstFgjTqA` (build 0.5.1,
  `searchQuery=python, maxPages=4, maxItems=40`, shipped RESIDENTIAL
  default) delivered 40 dataset rows for only 37 distinct `job_id`s — 3
  jobs (`2096544062622277429` "Python Developer for API",
  `2097908879856578200` "Python Workflow Automation Specialist",
  `2096857568662467307` "Python QA Engineer Role") were emitted twice,
  verbatim. `chargedEventCounts` was `{"actor-start": 1, "result": 40}`, so
  the customer paid `result` x 40 at $0.004 for 37 real jobs — double-billed
  for 3 of them (7.5% of the run). Root cause: Upwork's own result set
  shifts between page fetches (a new posting pushes listings down a page),
  so the same job legitimately appears on two consecutive pages — normal
  pagination drift, but nothing deduped it before the push/charge.
  `src/scraper.py`'s `run()` now keeps an in-run `seen_job_ids` set (an
  extension of REQ-9's own dedup key, which the README already promises:
  "`job_id` falls back to a stable content hash so leads still dedupe
  across scheduled re-runs" — this makes in-run behaviour match that
  promise) and never yields a `job_id` already emitted this run. Dedup
  happens inside the generator itself, *before* `main.py`'s
  `_scrape_and_push` ever sees the row, so both `Actor.push_data()` and the
  `result` charge see only distinct jobs — `_scrape_and_push`'s own
  `count=len(batch)` charge logic needed no change, because a batch can
  now never contain a repeat. The `seen_job_ids` set lives for the whole
  generator lifetime, not per-page or per-batch, so a repeat is caught even
  when the original copy was already flushed in an earlier
  `Actor.push_data()` batch — the case a naive per-batch-only dedup would
  miss.
  **`max_items` now caps DISTINCT delivered rows, not raw tiles seen** —
  the honest reading, since distinct rows are what the customer is billed
  for. A duplicate no longer consumes the budget; pagination keeps going
  (subject to `max_pages`) until the cap is met with real, distinct jobs.
  A fully-duplicate page now yields zero new rows, triggers zero extra
  `result` charge, and — because the existing `if batch:` empty-check in
  `_scrape_and_push` was already correct — never pushes an empty batch.
  Every suppressed duplicate logs at INFO (`suppressing duplicate
  job_id=... on page N`), plus a one-line per-run summary
  (`suppressed N duplicate job_id(s) this run`), so the behaviour is
  visible in a run log instead of silent.
  5 new tests (`tests/test_scraper.py`, `tests/test_main.py`), built on the
  REAL `parser.extract_tiles()` output of the real `page1_html` fixture
  capture (never a hand-typed guess at the tile payload shape): a repeated
  `job_id` across pages is emitted once; dedup survives a `BATCH_SIZE`
  boundary (the repeat is suppressed even though the original already left
  the batch in an earlier flush); a fully-duplicate second page adds no
  rows/charge/empty-push; `max_items` is satisfied by distinct rows across
  pages, not raw tiles. 123/123 tests pass; ruff + pyright clean.
  `_StartFee`/`charge_once()` (v0.5) is unchanged: a zero-row run still
  charges nothing, and `actor-start` still fires exactly once, from the
  first batch that reaches the dataset.

### 0.5 — 2026-09-18

- **Fix (billing correctness): `actor-start` was charged at boot, before
  the browser ever launched.** `main()` called
  `await _charge(EVENT_ACTOR_START, count=1)` immediately after
  `_validate_input`, so every run that then failed to clear Upwork's
  Cloudflare challenge — measured cloud clearance ~87-90% (spec.md's
  2026-09-17 recon: 9/10 on the proven RESIDENTIAL+`geoip=True` config) —
  still billed the customer $0.20 for zero delivered rows. That is the
  same shape that got `vrbo-vacation-rentals-scraper` delisted at 17
  runs (its CHANGELOG 0.3.0): a one-way-door defect on a public,
  monetized Actor.
  Fixed by porting the `_StartFee` pattern from
  `tiktok-ad-library-scraper` (commit f347bce9, itself ported from
  `google-ads-transparency` v0.5, verified in production this morning —
  a failed tiktok run reported `chargedEventCounts {'actor-start': 0}`).
  `_StartFee.charge_once()` is idempotent and now fires from
  `_scrape_and_push`, right after the FIRST `Actor.push_data()` batch
  that actually reaches the dataset — never from boot. A zero-row run
  (bootstrap failure, mid-run block, or a genuine zero-match search)
  charges nothing at all; a run spanning any number of batches still
  charges `actor-start` exactly once. `result` charging is unchanged.
  Input validation (REQ-1) still runs before anything else, and
  bootstrap/blocked failures still fail loud (REQ-2a/REQ-2b) — neither
  behaviour was touched.
  New pytest coverage in `tests/test_main.py`: zero-row run charges
  nothing, a many-batch run charges `actor-start` exactly once,
  `_StartFee.charge_once()` is idempotent, a bootstrap failure charges
  nothing even after the fee object exists, and a PPE charge failure
  never kills the run. `scripts/verify_charge_order.py
  upwork-job-leads-scraper` passes (`no-start-charge` — the boot-time
  charge call site is gone).

### 0.4 — 2026-09-18

- **Fix (correctness): `source_url` shipped Upwork's own unstripped-title
  markup and our crawl-tracking query string.** The prior 0.3.x fix
  absolutised the tile href (`/jobs/...` → `https://www.upwork.com/jobs/...`)
  but kept its slug segment byte-for-byte. Upwork wraps the matched search
  keyword in `<span class="highlight">` inside the title with no whitespace
  guarantee, and slugifies that SAME unstripped string into the href — so
  every row's "canonical" URL actually read
  `https://www.upwork.com/jobs/span-class-highlight-Python-span-Dev_~022.../
  ?referrer_url_path=/nx/search/jobs/`. Measured 30/30 in cloud run
  `LrfpU9bXEDp2GzJJo`.
  `src/parser/tiles.py` no longer builds `source_url` from the href slug at
  all. It now builds a clean canonical URL from the numeric job id alone —
  `data-ev-job-uid` prefixed `"02"` (REQ-9's already-reliable primary
  identifier; confirmed identical to the href's own `~<digits>` tail across
  every tile in a real 2026-09-18 capture), or, only when that attribute is
  absent, the id parsed via regex straight out of the href's `~<digits>`
  tail — never the slug text. Neither path can carry markup or a query
  string, because neither path touches the slug/query at all.
  `_absolutise()` is removed; it no longer has a caller.
- **Real-fixture regression coverage.** Added
  `tests/fixtures/html_tiles_live_highlight.html` — a byte-for-byte real
  capture (Camoufox + Apify RESIDENTIAL + `geoip=True`, per
  `reference-hand-authored-fixtures-encode-fiction`, no hand-authored HTML)
  of `upwork.com/nx/search/jobs/?q=python`'s 10-tile results block, including
  the real `<span class="highlight">` markup and `data-ev-job-uid`
  attributes. `test_extract_tiles_live_capture_source_url_has_no_markup_or_
  query` asserts none of its 10 real `source_url`s contain `span`, `class`,
  `highlight`, `<`, or a `?` query string.
- **Adjacent parse-completeness fix (checked, not guessed).** Sampled
  `posted_at`/`price_min`/`price_max` gaps flagged by the daily report
  against the same real capture: `posted_at` nulls trace to "Posted last
  week" / "Posted last month" phrasing the parser deliberately doesn't
  cover (REQ-10, unchanged, working as designed) — genuinely
  absent-in-source, not a miss. `price_min`/`price_max` nulls on a bare
  "Hourly" label (no visible rate range anywhere on the tile, confirmed in
  the raw markup) are also genuinely absent. But `price_type` going `None`
  on that same bare "Hourly" WAS a real miss — "Fixed price" (no numbers
  either) already resolved to `("fixed", None, None)`, so bare "Hourly"
  now resolves to `("hourly", None, None)` for the same reason
  (`src/parser/normalize.py`).
- Local verification: 114/114 tests pass, ruff + pyright clean, plus a real
  local `apify run` against `tests/fixtures/input.qa.json` (RESIDENTIAL +
  Camoufox `geoip=True`, cleared on attempt 1) — all 10 dataset rows sampled
  clean `source_url`s with no markup/query, matching the fixture assertion.
  Cloud QA (2 SUCCEEDED runs ≥1h apart, per the anti-bot-class publish gate)
  pending — this Actor stays SHELVED in `scripts/publish_actors.py` until a
  human un-shelves it after reviewing clean cloud-dataset samples.

### 0.3 — 2026-09-17

- **Fix (correctness): a block was being reported as a genuine empty search.**
  Cloud QA run `b2BUmvUQPimWcKicS` (build 0.2.1) finished SUCCEEDED with 0
  rows for `q=python` — a term proven to return 127+ real matches. Bootstrap
  attempt 1 correctly logged `stall` ("Blocked - Upwork"); attempt 2 hit a
  not-yet-catalogued block-page shape that matched none of
  `STALL_TITLE_MARKERS`/`DENIED_TITLE_MARKER`/`CHALLENGE_BODY_MARKERS`, and
  `classify_page`'s old fallback treated "no known marker matched" as
  `cleared` (REQ-8's genuine-empty case) — so it never reached bootstrap
  attempt 3 and shipped an empty-but-green run.
  `src/parser/blocking.py` now requires a POSITIVE marker
  (`data-test="job-tile-list"`, the real results container, present on every
  genuine render regardless of match count) before a zero-tile page is
  classified `cleared`; absent that marker it now classifies `stall`
  (retried, and eventually a loud `UpworkBootstrapError` / non-zero exit —
  never a silent 0-row success).
  `src/browser.py`'s `attempt_bootstrap()` also no longer short-circuits to
  `cleared` the instant `wait_for_selector` resolves — a coincidental match
  of the broad `TILE_SELECTOR` against a non-tile element could previously
  skip classification entirely. Both the timeout path and the
  selector-resolved path now funnel through one `_classify_current_page()`
  step with a REAL tile count from the actual extractor, so the retry loop's
  full `MAX_BOOTSTRAP_ATTEMPTS=3` budget is honored whenever the page is
  actually still challenged.
- Reach was NOT independently re-verified past this fix (see
  `docs/specs/upwork-job-leads-scraper/notes.md` for the cloud-QA-gated
  status) — the fix's own correctness is covered by unit tests
  (`test_parser_blocking.py`, `test_browser.py`) reproducing the exact
  unrecognized-block-shape scenario from the QA log.

### 0.2 — 2026-09-17

- Real Camoufox-driven implementation replaces the 0.1 scaffold stub.
  `src/browser.py` exposes Camoufox/Playwright primitives plus
  `attempt_bootstrap()` (waits on the tile selector — `~45s` budget —
  instead of a fixed sleep, then classifies cleared/stall/denied from
  content, never HTTP status alone).
- `src/client.py`'s `UpworkClient` implements the bootstrap-retry state
  machine (REQ-2/2a/2b, `MAX_BOOTSTRAP_ATTEMPTS=3`, a fresh browser context
  per attempt, fails loud and distinctly on a hard denial vs. an exhausted
  stall) and reuses the ONE cleared context for the rest of the run —
  pagination's clearance cookie persists per-context, so this Actor pays the
  probabilistic clearance cost once, not once per page (structural
  divergence from `ziprecruiter-jobs-scraper`'s per-page relaunch). Mid-run
  degradation (REQ-5) gets its own smaller rotation budget
  (`MID_RUN_ROTATION_ATTEMPTS=2`); ordinary transport errors get their own
  exponential-backoff retry (REQ-13, base 2s / cap 30s / 5 attempts).
- `src/parser/` — `blocking.classify_page()` (pure, shared by bootstrap and
  mid-run paths) distinguishes a genuine zero-match search from a stall/
  denial from content alone (REQ-3/REQ-8); `tiles.extract_tiles()` walks the
  parsed tree once per node (a `tree.css()` comma-union call was found to
  double-count nodes matching both selector branches — fixed before
  shipping); `normalize.py` best-effort parses posted-age and price text.
- `src/models.py` — `ActorInput`/`ResultRow` (Pydantic v2, ADR-0004).
  **Load-bearing default**: `proxy_configuration` now forces
  `apifyProxyGroups: ["RESIDENTIAL"]` — the 2026-09-17 cloud recon proved
  bare Apify runner IP and RESIDENTIAL-without-`geoip` both score 0/N
  against this target, while RESIDENTIAL + Camoufox `geoip=True` scores
  9/10. `.actor/input_schema.json`'s `proxyConfiguration` default/prefill
  and `maxPages` prefill (bumped 2 → 3) match this correction.
- `src/scraper.py` — pagination loop honoring `max_pages`/`max_items`
  (REQ-6), zero-tiles-means-end-of-results (REQ-4), `job_id` via a real
  tile identifier or a stable content hash (REQ-9), and REQ-11's
  `ResultRow.model_dump(mode="json")` emit path — no raw dict rows.
- `src/main.py` — REQ-1 input validation before any browser launch,
  distinct fail-loud paths for a bootstrap failure (stall vs. denied),
  mid-run block, and a genuine fetch failure, vs. REQ-8's honest
  zero-rows-is-not-a-failure success path.
- `tests/` — 103 unit tests (models, parser × 3, browser, client, scraper,
  main) plus HTML fixtures for tiles/empty-results/stall/denied pages.
  `tests/fixtures/input.qa.json` uses `maxPages: 3, maxItems: 10` (REQ-15)
  so cloud QA actually exercises the session-reuse/pagination path, not
  just page 1.

### 0.1 — 2026-09-14

- Scaffolded skeleton (actor-scaffolder). Boots, reads input, pushes one
  clearly-marked placeholder dataset row, exits cleanly. Real Camoufox-driven
  bootstrap-retry / pagination / parsing pipeline pending
  (`crawlee-developer`, per `docs/specs/upwork-job-leads-scraper/tasks.md`).