Upwork Job Leads Scraper
Pricing
from $4.00 / 1,000 result emitteds
Scrape Upwork's public job-search results by keyword. Get normalized rows with title, posted-age, price type/range, and experience level — built for freelancers monitoring for fresh leads. No login required. A freelance jobs scraper.
job_ids were double-billed
within a single run. Cloud run CK7RlxycwstFgjTqA (build 0.5.1,
searchQuery=python, maxPages=4, maxItems=40, shipped RESIDENTIAL
default) delivered 40 dataset rows for only 37 distinct job_ids — 3
jobs (2096544062622277429 "Python Developer for API",
2097908879856578200 "Python Workflow Automation Specialist",
2096857568662467307 "Python QA Engineer Role") were emitted twice,
verbatim. chargedEventCounts was {"actor-start": 1, "result": 40}, so
the customer paid result x 40 at $0.004 for 37 real jobs — double-billed
for 3 of them (7.5% of the run). Root cause: Upwork's own result set
shifts between page fetches (a new posting pushes listings down a page),
so the same job legitimately appears on two consecutive pages — normal
pagination drift, but nothing deduped it before the push/charge.
src/scraper.py's run() now keeps an in-run seen_job_ids set (an
extension of REQ-9's own dedup key, which the README already promises:
"job_id falls back to a stable content hash so leads still dedupe
across scheduled re-runs" — this makes in-run behaviour match that
promise) and never yields a job_id already emitted this run. Dedup
happens inside the generator itself, before main.py's
_scrape_and_push ever sees the row, so both Actor.push_data() and the
result charge see only distinct jobs — _scrape_and_push's own
count=len(batch) charge logic needed no change, because a batch can
now never contain a repeat. The seen_job_ids set lives for the whole
generator lifetime, not per-page or per-batch, so a repeat is caught even
when the original copy was already flushed in an earlier
Actor.push_data() batch — the case a naive per-batch-only dedup would
miss.
max_items now caps DISTINCT delivered rows, not raw tiles seen —
the honest reading, since distinct rows are what the customer is billed
for. A duplicate no longer consumes the budget; pagination keeps going
(subject to max_pages) until the cap is met with real, distinct jobs.
A fully-duplicate page now yields zero new rows, triggers zero extra
result charge, and — because the existing if batch: empty-check in
_scrape_and_push was already correct — never pushes an empty batch.
Every suppressed duplicate logs at INFO (suppressing duplicate job_id=... on page Nsuppressed N duplicate job_id(s) this run), so the behaviour is
visible in a run log instead of silent.
5 new tests (tests/test_scraper.py, tests/test_main.py), built on the
REAL parser.extract_tiles() output of the real page1_html fixture
capture (never a hand-typed guess at the tile payload shape): a repeated
job_id across pages is emitted once; dedup survives a BATCH_SIZE
boundary (the repeat is suppressed even though the original already left
the batch in an earlier flush); a fully-duplicate second page adds no
rows/charge/empty-push; max_items is satisfied by distinct rows across
pages, not raw tiles. 123/123 tests pass; ruff + pyright clean.
_StartFee/charge_once() (v0.5) is unchanged: a zero-row run still
charges nothing, and actor-start still fires exactly once, from the
first batch that reaches the dataset.actor-start was charged at boot, before
the browser ever launched. main() called
await _charge(EVENT_ACTOR_START, count=1) immediately after
_validate_input, so every run that then failed to clear Upwork's
Cloudflare challenge — measured cloud clearance ~87-90% (spec.md's
2026-09-17 recon: 9/10 on the proven RESIDENTIAL+geoip=True config) —
still billed the customer $0.20 for zero delivered rows. That is the
same shape that got vrbo-vacation-rentals-scraper delisted at 17
runs (its CHANGELOG 0.3.0): a one-way-door defect on a public,
monetized Actor.
Fixed by porting the _StartFee pattern from
tiktok-ad-library-scraper (commit f347bce9, itself ported from
google-ads-transparency v0.5, verified in production this morning —
a failed tiktok run reported chargedEventCounts {'actor-start': 0}).
_StartFee.charge_once() is idempotent and now fires from
_scrape_and_push, right after the FIRST Actor.push_data() batch
that actually reaches the dataset — never from boot. A zero-row run
(bootstrap failure, mid-run block, or a genuine zero-match search)
charges nothing at all; a run spanning any number of batches still
charges actor-start exactly once. result charging is unchanged.
Input validation (REQ-1) still runs before anything else, and
bootstrap/blocked failures still fail loud (REQ-2a/REQ-2b) — neither
behaviour was touched.
New pytest coverage in tests/test_main.py: zero-row run charges
nothing, a many-batch run charges actor-start exactly once,
_StartFee.charge_once() is idempotent, a bootstrap failure charges
nothing even after the fee object exists, and a PPE charge failure
never kills the run. scripts/verify_charge_order.py upwork-job-leads-scraperno-start-charge — the boot-time
charge call site is gone).source_url shipped Upwork's own unstripped-title
markup and our crawl-tracking query string. The prior 0.3.x fix
absolutised the tile href (/jobs/... → https://www.upwork.com/jobs/...)
but kept its slug segment byte-for-byte. Upwork wraps the matched search
keyword in <span class="highlight"> inside the title with no whitespace
guarantee, and slugifies that SAME unstripped string into the href — so
every row's "canonical" URL actually read
https://www.upwork.com/jobs/span-class-highlight-Python-span-Dev_~022.../ ?referrer_url_path=/nx/search/jobs/LrfpU9bXEDp2GzJJo.
src/parser/tiles.py no longer builds source_url from the href slug at
all. It now builds a clean canonical URL from the numeric job id alone —
data-ev-job-uid prefixed "02" (REQ-9's already-reliable primary
identifier; confirmed identical to the href's own ~<digits> tail across
every tile in a real 2026-09-18 capture), or, only when that attribute is
absent, the id parsed via regex straight out of the href's ~<digits>
tail — never the slug text. Neither path can carry markup or a query
string, because neither path touches the slug/query at all.
_absolutise() is removed; it no longer has a caller.tests/fixtures/html_tiles_live_highlight.html — a byte-for-byte real
capture (Camoufox + Apify RESIDENTIAL + geoip=True, per
reference-hand-authored-fixtures-encode-fiction, no hand-authored HTML)
of upwork.com/nx/search/jobs/?q=python's 10-tile results block, including
the real <span class="highlight"> markup and data-ev-job-uid
attributes. test_extract_tiles_live_capture_source_url_has_no_markup_or_ querysource_urls contain span, class,
highlight, <, or a ? query string.posted_at/price_min/price_max gaps flagged by the daily report
against the same real capture: posted_at nulls trace to "Posted last
week" / "Posted last month" phrasing the parser deliberately doesn't
cover (REQ-10, unchanged, working as designed) — genuinely
absent-in-source, not a miss. price_min/price_max nulls on a bare
"Hourly" label (no visible rate range anywhere on the tile, confirmed in
the raw markup) are also genuinely absent. But price_type going None
on that same bare "Hourly" WAS a real miss — "Fixed price" (no numbers
either) already resolved to ("fixed", None, None), so bare "Hourly"
now resolves to ("hourly", None, None) for the same reason
(src/parser/normalize.py).apify run against tests/fixtures/input.qa.json (RESIDENTIAL +
Camoufox geoip=True, cleared on attempt 1) — all 10 dataset rows sampled
clean source_urls with no markup/query, matching the fixture assertion.
Cloud QA (2 SUCCEEDED runs ≥1h apart, per the anti-bot-class publish gate)
pending — this Actor stays SHELVED in scripts/publish_actors.py until a
human un-shelves it after reviewing clean cloud-dataset samples.b2BUmvUQPimWcKicS (build 0.2.1) finished SUCCEEDED with 0
rows for q=python — a term proven to return 127+ real matches. Bootstrap
attempt 1 correctly logged stall ("Blocked - Upwork"); attempt 2 hit a
not-yet-catalogued block-page shape that matched none of
STALL_TITLE_MARKERS/DENIED_TITLE_MARKER/CHALLENGE_BODY_MARKERS, and
classify_page's old fallback treated "no known marker matched" as
cleared (REQ-8's genuine-empty case) — so it never reached bootstrap
attempt 3 and shipped an empty-but-green run.
src/parser/blocking.py now requires a POSITIVE marker
(data-test="job-tile-list", the real results container, present on every
genuine render regardless of match count) before a zero-tile page is
classified cleared; absent that marker it now classifies stall
(retried, and eventually a loud UpworkBootstrapError / non-zero exit —
never a silent 0-row success).
src/browser.py's attempt_bootstrap() also no longer short-circuits to
cleared the instant wait_for_selector resolves — a coincidental match
of the broad TILE_SELECTOR against a non-tile element could previously
skip classification entirely. Both the timeout path and the
selector-resolved path now funnel through one _classify_current_page()
step with a REAL tile count from the actual extractor, so the retry loop's
full MAX_BOOTSTRAP_ATTEMPTS=3 budget is honored whenever the page is
actually still challenged.docs/specs/upwork-job-leads-scraper/notes.md for the cloud-QA-gated
status) — the fix's own correctness is covered by unit tests
(test_parser_blocking.py, test_browser.py) reproducing the exact
unrecognized-block-shape scenario from the QA log.src/browser.py exposes Camoufox/Playwright primitives plus
attempt_bootstrap() (waits on the tile selector — ~45s budget —
instead of a fixed sleep, then classifies cleared/stall/denied from
content, never HTTP status alone).src/client.py's UpworkClient implements the bootstrap-retry state
machine (REQ-2/2a/2b, MAX_BOOTSTRAP_ATTEMPTS=3, a fresh browser context
per attempt, fails loud and distinctly on a hard denial vs. an exhausted
stall) and reuses the ONE cleared context for the rest of the run —
pagination's clearance cookie persists per-context, so this Actor pays the
probabilistic clearance cost once, not once per page (structural
divergence from ziprecruiter-jobs-scraper's per-page relaunch). Mid-run
degradation (REQ-5) gets its own smaller rotation budget
(MID_RUN_ROTATION_ATTEMPTS=2); ordinary transport errors get their own
exponential-backoff retry (REQ-13, base 2s / cap 30s / 5 attempts).src/parser/ — blocking.classify_page() (pure, shared by bootstrap and
mid-run paths) distinguishes a genuine zero-match search from a stall/
denial from content alone (REQ-3/REQ-8); tiles.extract_tiles() walks the
parsed tree once per node (a tree.css() comma-union call was found to
double-count nodes matching both selector branches — fixed before
shipping); normalize.py best-effort parses posted-age and price text.src/models.py — ActorInput/ResultRow (Pydantic v2, ADR-0004).
Load-bearing default: proxy_configuration now forces
apifyProxyGroups: ["RESIDENTIAL"] — the 2026-09-17 cloud recon proved
bare Apify runner IP and RESIDENTIAL-without-geoip both score 0/N
against this target, while RESIDENTIAL + Camoufox geoip=True scores
9/10. .actor/input_schema.json's proxyConfiguration default/prefill
and maxPages prefill (bumped 2 → 3) match this correction.src/scraper.py — pagination loop honoring max_pages/max_items
(REQ-6), zero-tiles-means-end-of-results (REQ-4), job_id via a real
tile identifier or a stable content hash (REQ-9), and REQ-11's
ResultRow.model_dump(mode="json") emit path — no raw dict rows.src/main.py — REQ-1 input validation before any browser launch,
distinct fail-loud paths for a bootstrap failure (stall vs. denied),
mid-run block, and a genuine fetch failure, vs. REQ-8's honest
zero-rows-is-not-a-failure success path.tests/ — 103 unit tests (models, parser × 3, browser, client, scraper,
main) plus HTML fixtures for tiles/empty-results/stall/denied pages.
tests/fixtures/input.qa.json uses maxPages: 3, maxItems: 10 (REQ-15)
so cloud QA actually exercises the session-reuse/pagination path, not
just page 1.crawlee-developer, per docs/specs/upwork-job-leads-scraper/tasks.md).