Upwork Jobs Scraper | 30+ Filters & Duplicate-Free Runs
Pricing
from $2.00 / 1,000 results
Scrape Upwork jobs with title, FULL description, budget, skills and project length. 30+ filters: budget floors across hourly AND fixed, keyword include/exclude, client country, experience level, freshness window. Monitoring mode skips jobs earlier runs returned, so you never pay twice.
Newest first. Dates are the day the work landed.
main.py charged two pay-per-event units by hand, job-result on every pushed row and job-detail
on every detail fetch. Neither event has ever existed. The published Actor is monetized on
Apify's two built-in events, confirmed against the Store API:
declared on tqm/upwork-jobs-scraper | fires |
|---|---|
apify-actor-start | once per run, platform-side |
apify-default-dataset-item | every push_data, platform-side |
So Actor.charge(event_name="job-result") raised on every single row, and _charge() swallowed it:
No revenue was lost - apify-default-dataset-item was already charging the intended $0.0025 for
each of those rows, which is why nothing looked wrong. The bug was the reverse of a leak:
job-result in the Console. Doing so would have added
a second per-row charge on top of the built-in one and billed every buyer twice. All three docs
now say so explicitly.job-detail was already unreachable. N1 stripped scrapeJobDetails from the sold schema, so
no buyer can trigger the detail path at all. Verified: the Store build exposes 34 inputs, the
internal build 40 - exactly the six N1 removes.except Exception: pass on a billing call, which is what kept it invisible for a
month. Removed with its only caller.Billing is now entirely platform-side and main.py contains no Actor.charge() call, with a
comment at the old constant site saying why one must not be re-added.
347 tests, unchanged - nothing referenced the removed helper.
scrapeJobDetails was off by default and documented as expensive. That is not the same as safe. It
was a buyer-settable switch, and the buyer does not pay the platform cost - we do, out of the
80% revenue share. Measured on the same actor, the same day:
| tier | rows | platform cost | revenue at $2.50/1k |
|---|---|---|---|
card (scrapeJobDetails: false) | 50 | $0.019 | $0.100 |
detail (true) | ~1 | $1.28 | $0.002 |
A 640x loss on one run. Apify's loss protection floors a losing month at $0 rather than a debt, so the real exposure is not a bill - it is that any buyer could zero out the actor's entire monthly earnings with one checkbox.
And it is not a Cloudflare problem. The $1.28 figure is the Turnstile-blocked worst case, but the tier is one residential browser page-load per row by design. Even at a 100% success rate that is an order of magnitude above $2.50/1,000. There is no unblocked version of this that pays.
Stripped from the Store copy by scripts/store_schema.py, which the mirror workflow runs in the
runner's checkout just after the README swap:
| field | why it had to go with it |
|---|---|
scrapeJobDetails | the switch itself |
detailAttempts | tunes retries on a path buyers can no longer take |
paymentVerifiedOnly | reads client.paymentVerified; drops rows whose value is unknown, so card-only it returns zero rows |
minClientSpendUsd | same - drops on unknown |
minClientHires | same - drops on unknown |
excludeClientCountries | needs client.country, which no card carries. Fails open rather than closed, so it silently narrowed nothing at all |
clientCountries stays and is the one client filter that works, because Upwork applies it
server-side as GraphQL location (verified additively 2026-07-31: US=444, UK=78, both=522). It
makes runs cheaper as well as narrower.
Why a CI transform and not an edit to the committed schema. Both accounts build from this one
file. moisecristian2/upwork-jobs-scraper is what TQM's nightly calls, and it sends
scrapeJobDetails: false explicitly (apify_ingest.py, with a documented trigger for turning it
back on). Deleting the field from the repo would have put an undeclared key into a validated input
on a production run every night - the first thing the removal would have broken is the pipeline
this actor was built for. A removal LIST also cannot drift the way a second copy of the schema
would: a renamed input raises rather than shipping.
src/ is unchanged apart from one warning string: the maxProposals message no longer tells buyers
to set an input that no longer exists.
347 tests green (334 + 13 covering the transform). One of the new tests found a real bug before it shipped: the dataset transform appended its "not populated" sentence again on a second run.
Neither is caused by the change above; both were found by checking every claim in the listing against the code rather than trusting the previous copy.
proposalsMin is null on a default run. Upwork does not expose proposal counts through the
JSON search endpoint, which is the default path - tests/test_graphql_map.py:129 asserts exactly
this. The listing advertised the field flatly, showed "proposalsMin": 5 in its output sample,
and put maxProposals in the sample input. A buyer would have paid for a run, got nulls, and
been right to call it broken. Now stated on the field, the filter and the sample.description was documented as a truncated card snippet. It has not been one since the JSON
path landed - that path returns the complete text and the full skill list on every row. The
dataset schema was understating the actor's best feature.Also corrected in .actor/dataset_schema.json: client, detailScraped, expiresAt,
interviewingCount, invitesSent, unansweredInvites and lastViewedByClient now say they are
not populated, instead of "Detail scraping only" - a condition a Store buyer can no longer create.
The fields stay declared, not deleted, because the actor still emits them as nulls and this
repo has already shipped a dataset schema that rejected its own output.
The Actor had been returning zero rows and TIMED-OUT at its 30-minute cap for three nights running (2026-09-02/03/04) in the TQM nightly pipeline. Diagnosed from the run log:
The token fetch failed, the run fell back to the browser, and the browser hit the Cloudflare "managed" Turnstile at ~5 min per page until the cap killed it.
Token acquisition is a PROXY-IP problem, not a fingerprint problem. Measured through Apify RESIDENTIAL over 12 fresh sessions per fingerprint:
| fingerprint | tokens |
|---|---|
| chrome131 | 5/12 (41%) |
| chrome136 | 5/12 (41%) |
| safari18_0 | 4/12 (33%) |
Roughly 40% of residential IPs yield a token; the rest are burned. The client tried every fingerprint against one identity, so it failed about 60% of the time and took the whole JSON path with it. Rotating to a fresh identity fixes it - 6/6 trials got a token within 4 attempts (1, 2, 3, 3, 3, 4).
_ensure_token now loops proxy identities on the outside, fingerprints on the inside
(PROXY_ROTATIONS = 6, ~95% at 40% per-IP), and main.py passes a proxy_url_factory. One
identity is still held for the life of a token - it is bound to the egress IP - so only
acquisition rotates. Without a factory the client behaves exactly as before.
[!warning] The 2026-07-31 claim that "Chrome impersonation does not work" was drawn from ONE sample per fingerprint, and the comment told future readers chrome was "deliberately excluded". At 12 samples chrome131 measures better than safari18_0. A one-shot-per-cell result is indistinguishable from noise at these rates - sample the cell more than once before excluding a value on the strength of it.
Verified end to end, 5 independent runs of the real client through Apify RESIDENTIAL: 5/5 obtained a token and returned rows, in 3-6 seconds each, against production's 30-minute timeout and zero rows.
Suite 330 → 334: rotation on a burned identity, single-identity back-compat, giving up after
PROXY_ROTATIONS, and an exhausted pool raising VisitorSearchError rather than StopIteration.
test_uses_the_first_impersonation_that_yields_a_token now asserts IMPERSONATIONS[0] instead of
the literal safari18_0 - pinning the literal made it fail for the right behaviour.
Two follow-ups from the day's findings, both closed.
jsonFallbacks no longer hides behind a clean SUCCEEDED. A run that lost the cheap path used to
end with Done. 10 jobs scraped. - identical to one that never needed the browser. The completion
message now names it, and requireJsonSearch (default off) lets a buyer who chose this Actor for
its cost profile fail instead of being silently upgraded to the ~200x path. Off by default because
for most buyers a browser-served result set still beats no result set. New pure
completion_message(), 9 tests in tests/test_json_fallback.py.
clientCountries - IMPROVEMENT-PLAN item 11, fixed after measuring first. The original write-up
offered "a small alias table or a pycountry dependency" and said the choice needed a wider sample.
Sampling 35 live detail pages across four queries settled it:
| Rendering | Count |
|---|---|
Country name (United States, Egypt, Canada) | 29 |
ISO-3 code (USA, EGY, EST, IRL, SGP, CHN) | 6 (17%) |
The decisive part is not the 17% - it is that the same country appears both ways in one sample:
United States and USA, Egypt and EGY. Any country can arrive either way, so a partial
alias table would have been the same silent-drop bug one layer down.
filters.canonical_country() now resolves both the buyer's input and the row's value to ISO
alpha-3 via pycountry, matching by equality rather than substring - which also fixes the
independent bug that "india" in "british indian ocean territory" was true. pycountry is pure
Python with no dependencies of its own and resolved cleanly against the load-bearing apify /
scrapling pins, checked with uv pip install --dry-run before adding.
A small _COUNTRY_ALIASES table remains for colloquial names ISO does not carry at all - UK,
Russia, Turkey (ISO lists Türkiye), UAE, Holland, Ivory Coast. Buyers type those; Upwork
never does. Bare Korea is deliberately unmapped, because a visible client_country_not_matched
beats a confident guess at the wrong peninsula.
The severity was worse than first recorded. The initial note said a buyer filtering for China lost
every Chinese client. True - but USA is among the codes, so a buyer filtering for the United
States, the platform's largest client market, was losing an unknown share of rows on an advertised
filter with no reason given.
17 tests in tests/test_country_matching.py, plus an end-to-end assertion in test_fixtures.py
against the real CHN row. Suite 302 → 330, green.
Commercialisation Phase 1, item 4. Version 0.1 is now GIT_REPO pinned to
git@github.com:Atredies/upwork-scraper.git#main, with .github/workflows/apify-build.yml matching
the eight sibling Actors. This was the last Actor on SOURCE_FILES, and the hand-uploaded source is
precisely what let git and the platform drift ~1,450 lines apart.
The first build failed: Could not read from remote repository. The repo's Apify deploy key was
stale - titled Apify Deployment Key (moisecristian2/upwork-scraper), from an earlier incarnation,
and not the key this Actor now presents. The Actor's live key (GET /v2/acts/{id} → deploymentKey)
was added and build 0.1.20 SUCCEEDED from main. Worth knowing that the key is per Actor and
survives nothing: if an Actor is recreated, its old key stays on the repo looking plausible.
[!warning] The platform runs exposed what local runs could not: the JSON search path is a coin flip through the datacenter proxy
Five runs, identical input (
maxItems: 10, card-only,BUYPROXIES94952):
Run JSON path Fingerprint Cost Time Nfj3zahFOq6rr1VbV❌ fell back - $0.0271 119 s 7OwecDa7Lpe8LnIJz✅ worked safari17_0$0.0087 38 s lRRAjFUDRlkpFQuj5❌ fell back - $0.0139 60 s Qgoha3Sg435twM9C1✅ worked safari17_0$0.0105 46 s V2noInT0WoyZC7MlG❌ fell back - $0.0190 83 s 2 of 5, and both successes came from
safari17_0- the last fingerprint in the ladder. On residential it worked cleanly: runlOlMQ4WIcjBHPr62v, 100 rows,jsonFallbacks: 0, $0.072/1000. The account has zero residential capacity, which is why these used datacenter.Every one of the five reported
SUCCEEDED. Identical input, $0.0087 to $0.0271 - a 3.1x spread - decided purely by which path the run happened to get. For a pay-per-event listing the problem is not that the fallback is expensive, it is that cost per run is unpredictable.Two earlier drafts of this entry got it wrong from single runs, which is itself the finding: one run tells you nothing here. The first said the cost advantage had "never been demonstrated on the platform" - wrong,
lOlMQ4WIcjBHPr62vis a 100-row production run with the browser never started. The second said datacenter hard-blocks the path - also wrong, it works 2 times in 5.Note the 3.1x is not the ~200x quoted between tiers elsewhere: at 10 rows container startup dominates both paths, and this repo's own rule is to measure per-1000 at >=100 rows. The per-path cost at scale is still unmeasured, and that is the number pricing needs. See
IMPROVEMENT-PLAN.mditem 12.
Read this before anything else in this file. The Apify actor had been deployed with
apify push without the source being committed, so Atredies/upwork-scraper@main and the platform
had diverged in both directions:
src/graphql_client.py, src/seen_store.py,
docs/LAUNCH-CRITERIA.md, three test modules, and the inputs jsonSearch,
deduplicateAgainstPreviousRuns, postedWithinHours, seenExpiryDays, seenStateKey.The deployed source is preserved verbatim on branch apify-deployed-snapshot. This entry is the
merge of the two, taking the deployed code as the base because it was ahead.
Two things came out of the merge that neither side had alone:
all_seen zero-row case was unreachable. The deployed _finish had a monitoring-mode
branch - "Nothing new, all N postings were already returned by earlier runs" - guarded by
elif seen.skipped: after elif state.parsed:. But _handle_card increments state.parsed
before the already-seen check, so an all-seen run always has parsed > 0 and always took the
filters branch instead, reporting "your filters removed all of them: {}" with an empty dict.
The merged diagnose_zero_rows() keys the filters branch on the breakdown dict rather than on
parsed, which makes the monitoring branch reachable for the first time.all_seen must not fail the run. The git side made every zero-row run fail; the deployed side
knew monitoring mode was a legitimate empty result. Both are right, so the diagnosis now returns
an is_failure flag: four causes fail, all_seen succeeds regardless of failOnZeroResults.
Without this, turning on deduplicateAgainstPreviousRuns would have paged the buyer every quiet
hour of their schedule._warn_about_unservable_filters() was also folded into the pure input_warnings(raw, cfg), which
now carries all three caveats - the JSON path's maxProposals, plus projectLength and
hourlyRateMin/Max from the URL-path verification - and is unit-tested rather than log-only.
Suite: 234 (deployed) → 302, green. The fixture suite passes against the deployed parser, and
caught the same Posted last week bug there.
Verified live, 2026-08-14, on the merged code, JSON path, browserStarted: false throughout:
| Run | Result |
|---|---|
Normal, hourlyRateMin + projectLength set | 5 rows; both input caveats logged at run start |
Monitoring, run 1 (maxItems: 50, sortBy: relevance) | 50 rows, seenLedgerSize: 50 |
| Monitoring, run 2, same input | cardsParsed: 50, skippedAsAlreadySeen: 50, filteredOut: {} → zeroRowReason: all_seen, exit 0 |
minClientSpendUsd: 999999999 | ZeroResultsError, exit 91 (FAILED), client_spend_unknown=50 |
The third row is the one that matters: it is the exact run that the deployed code would have reported as "filters removed all of them: {}", and that the git side would have failed.
The three entries below were written on the git side before the divergence was known, so their running test counts (146 → 159 → 195 → 207) are the git branch's numbering, not this one's. Their findings all hold; only the totals are superseded by the 302 above.
Commercialisation Phase 1, item 2 (IMPROVEMENT-PLAN.md §3). hourly_rate, amount and
duration_v3 were being sent on every search on the strength of an assumption. A param Upwork
silently drops produces an unfiltered result set that looks filtered, which is what
payment_verified did.
24 live fetches, one session, no blocks. All three survive the 307 redirect and demonstrably change
the result set - none is a second payment_verified, and no code change was needed to keep sending
them. But two do not mean what the input name implies:
hourly_rate matches OVERLAPPING ranges, not contained ones (job.max >= min AND job.min <= max):
| Query | Returned, among others |
|---|---|
hourly_rate=40-90 | $15-40, $25-47, $30-60 |
hourly_rate=100- | $25-100, $75-100 |
hourly_rate=-20 | $15-40, $10-25 |
hourly_rate=60-65 | $25-100, $30-60 |
So hourlyRateMin: 60 returns a $25-100/hr job. The input schema actively claimed the opposite -
"Only hourly jobs at or above this rate" - which is now corrected.
duration_v3 filters hourly jobs only and silently ignores fixed-price ones. The sharpest result
of the day:
| Query | Result |
|---|---|
t=1 (fixed), duration_v3=week vs =ongoing | 10/10 identical jobs |
t=0 (hourly), same pair | 0/10 overlap, every row matching its window |
amount is exact, on both bounds: 1000-5000 returned $1000/$1750/$2500 (read off the detail
pages, since fixed-price cards carry no amount) and -200 returned $10/$30/$50/$100. The $10 job it
correctly excluded is the posting saved as tests/fixtures/job-detail-fixed.html.gz.
Neither caveat is fixable post-fetch - roughly half of hourly cards carry no rate at all, and
fixed-price listings have no duration field whatsoever - so they are handled by telling the truth:
corrected input-schema descriptions, a README section, and main.input_warnings() logging a caveat
at run start when either filter is set. 12 tests in tests/test_input_warnings.py.
Suite 195 → 207, green.
Commercialisation Phase 1, item 3 (IMPROVEMENT-PLAN.md §4). tests/fake_dom.py supplies the
values it expects to see, so it can prove the selector ladder's logic but never that a selector
still matches real markup - which is exactly how the first live run returned zero rows past a
104/104 green suite.
tests/test_fixtures.py (33 tests) now runs the real scrapling/lxml engine over three pages
captured live on 2026-08-14, in one browser session, three fetches, no blocks:
| Fixture | Covers |
|---|---|
search.html | 10 cards - hourly and fixed, some with a card rate and some without |
job-detail.html | a client who joined that same day: the documented sparse case |
job-detail-fixed.html | full client block, plus the fixed-price amount/label pairing |
The two detail pages are deliberately opposite clients: the sparse one proves null-never-becomes-zero,
the rich one exercises client-spend, client-hires, client-hours and payment-verified - the
fields that are the actual product. tests/capture_fixtures.py is the re-capture recipe.
Import-guarded on scrapling; verified by shadowing the module, giving 33 clean skips rather than
33 errors. Stored gzipped in tests/fixtures/ - 3.5 MB raw, 489 KB compressed, byte-for-byte rather
than trimmed, since trimming risks deleting the very markup that drifts. tests/ is already in
.dockerignore, so none of it reaches the actor image.
Fixed, caught by the fixtures immediately: 1 of the 10 cards read Posted last week - no
digit and no "ago", so _POSTED_RE and _RELATIVE_RE both missed it and the row shipped
postedAt: null on a headline sortable field. Now resolved as 1 week, with the sibling last month
/ last day forms. yesterday is deliberately still unhandled - plausible, never observed, and
this repo does not ship formats it has not seen; a wrong guess invents a timestamp rather than
admitting ignorance. The pre-existing test that pinned yesterday as unparseable stays, with the
reasoning written down.
Found, not fixed: client-location renders either a country name (Sweden) or an ISO-3
code (CHN, confirmed in the raw markup). filters.py matches by lowercase substring, so
clientCountries: ["China"] silently drops every Chinese client - an advertised filter that
silently does nothing, which is the exact failure mode ARCHITECTURE rule 2 exists to prevent. It is
publish-blocking and the fix needs a decision (alias table vs a pycountry dependency against
load-bearing pins), so it is written up as IMPROVEMENT-PLAN.md item 11 rather than guessed at here.
Suite 159 → 195, green.
Commercialisation Phase 1, item 1. Portfolio-wide finding: all nine actors exit SUCCEEDED on an
empty dataset. Internally that is an annoyance; on a priced listing it means a buyer sees a
successful run with nothing in it and cannot tell a broken actor from an empty query.
diagnose_zero_rows() in main.py returns a (reason_code, message) pair. Reason codes:
blocked, filtered_out, no_cards, parsed_not_pushed._finish() writes the reason to RUN_STATS as zeroRowReason and raises ZeroResultsError,
which async with Actor turns into a failed run.failOnZeroResults, default true - the opt-out for scheduled monitoring where an
empty run is normal.tests/test_zero_results.py; suite 146 → 159, green.Three details worth recording because each was a decision, not an obvious step:
Actor.fail(). fail() calls Actor.__aexit__ itself and this code runs
inside async with Actor, so it would run the exit path twice. Raising is the context manager's
own documented failure route, and main.py already raises ValueError on empty search targets.RUN_STATS is written before the raise. A failed run is exactly when someone reads it.state.parsed increments
before the dedup check in _handle_card, so the first sighting of every jobId survives and
would be pushed. A test now pins that ordering, so if _handle_card changes, it fails loudly
rather than resurrecting a misleading message.Also fixed in passing: the filtered-out status message interpolated the raw Python dict, so buyers
saw {'client_spend_below_min': 34}. It is now client_spend_below_min=34, sorted for stable diffs.
Verified against live Upwork locally, both paths, 2026-08-14. Input: python developer, 1 page,
card-only, minClientSpendUsd: 999999999 to guarantee an empty result.
failOnZeroResults | Outcome | zeroRowReason |
|---|---|---|
true (default) | ZeroResultsError, exit code 91 (ERROR_USER_FUNCTION_THREW = FAILED) | filtered_out |
false | exit code 0, WARN in the log | filtered_out |
Status message both times: "Parsed 10 jobs but every one was removed by your filters
(client_spend_unknown=10). Loosen the filters and re-run." - and client_spend_unknown rather than
client_spend_below_min is the documented *_unknown behaviour for client filters without
scrapeJobDetails, working as intended.
Still owed a platform run, per CLAUDE.md - local runs have missed a missing browser binary and
a schema Apify rejected. The no_cards and blocked paths are covered by unit tests only; neither
is reproducible on demand against live Upwork.
The four things no other Upwork scraper on the Store has were built but invisible to a buyer scanning the page. Rewritten around them, and around real numbers rather than adjectives.
oauth2_global_js_token), live data (two rivals serve a ~6 h cache), tri-state nulls,
budgetMonthlyUsd, detailScraped, published coverage.client.totalSpentUsd
at 45% is the client genuinely having no history, not a parse failure - and the section says so,
along with the warning that combining sparse filters compounds the loss.RUN_STATS table - the run-summary counters were emitted and documented nowhere.sectionCaption sections (search / client data / Upwork-side /
result filters / freshness+monitoring / advanced) instead of 38 flat fields.compact dataset view.Verified on the platform that the schemas deploy and all four new features compose: a card-only run
with server-side jobType and clientCountries plus postedWithinHours returned 40 rows in 23 s
for $0.005, 40/40 hourly, 1 dropped as too_old, browserStarted: false, one GraphQL query.
Three levers, shipped together and measured against the identical input that produced the 0.1.9
baseline:
0.1.9 | 0.1.15 | ||
|---|---|---|---|
| Cost | $0.282 | $0.202 | -28% |
| Per 1000 results | $14.10 | $10.10 | |
| Wall clock | 9m13s | 6m35s | -29% |
| Residential traffic | 16.91 MB | 12.13 MB | -28% |
| Compute | 0.3073 CU | 0.2198 CU | -28% |
sessionsStarted | 1 | 1 | healthy |
1. network_idle off, wait for a selector instead. It was True at session construction, and
scrapling's _wait_for_page_stability() already waits for load and then domcontentloaded -
network_idle was a third wait for 500 ms of silence on a page that never goes quiet (analytics
beacons, chat widgets, lazy images). Playwright's own docs mark the state DISCOURAGED. Now waits for
an element that actually exists instead.
2. blocked_domains. scrapling supports hostname blocking and we were not using it.
disable_resources cannot help here: it blocks by resource type, and analytics ships as
script/xhr - the two types that must stay enabled or the Turnstile cannot run. ~35 hosts
(Google Analytics, Segment, Sentry, Hotjar, DoubleClick, ad networks) now never cross the proxy.
3. challenges.cloudflare.com is deliberately NOT in that list, and a test pins it there. The
Turnstile solver clicks the widget's computed bounding box; blocking its assets risks a
mis-laid-out widget and a failed solve, which costs a browser restart plus a fresh challenge - far
more than the bytes saved.
max_pages is now plumbed through Fetcher (scrapling's PagePool puts N tabs on one persistent
context, so they share the cf_clearance cookie and the challenge is still solved once) but
defaults to 1 and is not yet used concurrently. Deliberate: this run logged 9 blocked fetches
for 20 rows, so Upwork is already rate-limiting at one request at a time. Adding concurrency
before adaptive delay would very likely raise the block rate and cost more than it saves - see
IMPROVEMENT-PLAN §6/§7, which say the same thing in the other order.
Tests 228 -> 234.
The two things a scheduled buyer needs, and the two biggest input gaps against the competition (seven of twelve Store rivals have a freshness filter; four have cross-run dedup).
deduplicateAgainstPreviousRuns - remembers which jobIds previous runs emitted, in a named
key-value store scoped to this search, and skips them. A buyer running hourly on sortBy: recency
was previously re-paying for the same postings every hour; now they pay only for what is new. Four
competitors sell this as a cost saving rather than a feature, which is the correct framing.
Verified on the platform with two consecutive runs of the same search:
| Run | Pushed | skippedAsAlreadySeen | Ledger | Cost |
|---|---|---|---|---|
| 1 (seeds) | 15 | 0 | 15 | $0.005 |
| 2 | 15 new | 15 | 30 | $0.005 |
Design points, all load-bearing:
maxItems, proxy config and post-fetch filters deliberately do not.
Raising maxItems is the same ongoing search, and resetting there would re-bill the buyer for
everything - the exact failure this feature exists to prevent. seenStateKey overrides it.seenExpiryDays, default 30). Upwork postings go stale fast.postedWithinHours - post-fetch freshness bound. sortBy: recency orders results but does not
bound them. Postings whose age cannot be read are kept, never dropped (rule 3).
RUN_STATS gained skippedAsAlreadySeen, seenLedgerSize, seenScopeKey.
Tests 211 -> 228.
clientCountries is now pushed to Upwork as the GraphQL location variable instead of being applied
post-fetch. This fixes a real bug as much as it adds a feature: the post-fetch check rejects any row
whose client.country is unknown, and card-level rows have no country, so clientCountries
without scrapeJobDetails used to return zero rows.
Verified two ways, because a total that moves is not proof:
clientCountries: ["United States"] and scrapeJobDetails: true returned
8/8 rows with client.country: "United States" and an empty filteredOut - the post-detail
recheck, which still applies the full filter, rejected nothing.Card-only country-filtered run: 30 rows, 21s, $0.006.
Gotcha recorded in docs/UPWORK-DOM.md: full country names only. "US" returns 0 rows,
"United States" returns 444. A wrong name fails loudly rather than being ignored.
While testing the request schema, three more parameters were caught being accepted without
filtering - clientHires, skills, and most subtly durationV3, which changes the result set
without partitioning it (week returned "More than 6 months" rows; the four values total 3130
against a 1438 baseline). None are sent. projectLength therefore stays post-fetch.
Upwork hands any client a visitor_gql_token cookie on a plain GET / - no login, no
browser, no Cloudflare challenge - and that token authorises the same GraphQL search its
own logged-out UI uses. Search now goes through it by default (jsonSearch, default on),
and the browser is constructed lazily, only when something genuinely needs a rendered page.
Measured on the platform, residential proxy, same query:
| Build | Path | Rows | Time | Cost | Per 1000 |
|---|---|---|---|---|---|
0.1.6 | browser search + detail | 20 | 10m1s | $0.329 | $16,450 |
0.1.8 | JSON search, card-only | 100 | 38s | $0.007 | $70 |
0.1.9 | JSON search + browser detail | 20 | 9m13s | $0.282 | $14,100 |
The card-only tier is the one that moved, and it moved by ~235x: 3 HTTP requests and
247 KB of residential traffic for 100 rows, against 26 fetches and 20.74 MB for 20.
RUN_STATS on that run reads browserStarted: false, sessionsStarted: 0. Peak memory
117 MB, so it runs comfortably at 1024 MB. The detail tier only improved 14%, because it
is still one browser render per row - that is the honest shape of the win.
The JSON path is also richer at card level, which was not expected: it returns the
full description (median 1516 chars on a 100-row sample, against the card's truncated
snippet), the complete skill list (median 5), and publishTime as an exact ISO-8601
timestamp rather than "4 hours ago" resolved against scrape time. Two of those
previously required a detail fetch.
What it does not return: the client block. Query.jobPubDetails is scope-blocked
wholesale for the visitor token - verified field by field from two different IPs. So
scrapeJobDetails remains a browser operation and the client block stays the premium
tier. Public write-ups claiming otherwise are stale or were using a browser-session token.
Also landed:
jobType (lowercase scalar) and
contractorTier (PascalCase enums) genuinely narrow - proved by the per-value totals
summing to the unfiltered 1438. hourlyRate is accepted and silently ignored (a
40-150 request returned $12-20 rows), so it is not sent - the payment_verified trap
again. New design rule 8 in docs/ARCHITECTURE.md.safari18_0 and firefox135 pass. Every public
implementation of this technique uses impersonate="chrome", which is why they fail.workload regression the JSON path would have shipped. It is not in the
search payload and parse_detail() never set it, so a live 20-row detail run measured
workload at 0/20. parse_detail() now back-fills workload and duration from the
page, filling gaps only and never overwriting.maxProposals cannot be served on the JSON path (proposal counts are behind the scope
wall). Nothing is dropped - unknown values are kept, per rule 3 - but the run now
logs that the filter is not narrowing anything, rather than looking applied.RUN_STATS gained browserStarted, rowsFromJsonSearch, rowsFromBrowserSearch,
jsonFallbacks, graphqlQueries, graphqlBytes, graphqlImpersonation.startUrls deliberately stay on the browser path: a pasted URL may carry filters this
actor does not model, and translating it into GraphQL variables would silently drop them.Tests 146 → 186.
Every cost figure in this repo was read off the run list's rounded total rather than the platform's
own per-line breakdown, and two of the four were wrong. Corrected against
apify runs info <id> --verbose:
| Build | Was | Actually |
|---|---|---|
0.1.3 | $0.548 / $0.110 per result | $0.557 / $0.1114 |
0.1.6 | $0.290 / $0.0145 per result | $0.329 / $0.01645 |
0.1.4 and 0.1.5 totals were right. The headline number to quote is now $0.01645/result, or
$16.45 per 1000 results.
The more useful finding is the split, which nothing in this repo recorded: 59% of the bill is
residential proxy traffic, 41% is compute. Cost work aimed at wall-clock therefore addresses the
smaller half. Full per-line table under the 0.1.6 entry below.
Also recorded there: peak memory never exceeded 764 MB in any run, including one allocated 4096 MB.
Added docs/LAUNCH-CRITERIA.md - the measurable gate for going public and paid, with each criterion's
honest current status. Verdict against build 0.1.6: not ready, with four of eight areas unknown
rather than passing.
The two that matter most were not previously recorded anywhere: the default configuration has never
been run (maxItems: 100, maxPagesPerQuery: 3, scrapeJobDetails: false - every measurement so
far is a detail run of 3-20 items), and nothing has been run past 20 items. Also surfaced the
policy decisions nobody has made, starting with contactEmails extracting personal data from job text
under an EU-based seller.
Added docs/IMPROVEMENT-PLAN.md: a prioritised backlog where every item is grounded in something
observed on a live run, and each states why it matters, what to do, and how you would know it worked.
Top items are the two things blocking a sale - pay-per-event pricing, and measuring the card-only cost
path, which has never been run on the platform.
Added CLAUDE.md so future sessions read docs/ before touching code and update it as part of "done".
It also encodes the lesson from this repo's first three platform bugs: a green test suite and a working
local run are a weak signal here, so the platform run is the acceptance test, and emitted rows must be
checked rather than just the counters.
Two rounds of cost work, both driven by measurements on the platform rather than locally.
Round 1 - solve Cloudflare once, not per page. The per-URL StealthyFetcher.fetch() pattern
launched a fresh browser for every page and re-solved the Turnstile each time: 90-120s per page
behind a residential proxy, 5 results in 9m14s for $0.557, about $0.11/result. The actor now
holds one AsyncStealthySession for the whole run, so the cf_clearance cookie is earned once and
reused. solve_cloudflare is requested only when a challenge is expected, because the solver waits
for network idle even when there is nothing to solve. The fetcher went async at the same time -
scrapling's sync sessions bind Playwright to their creating thread, so a long-lived one driven from
asyncio.to_thread is a latent crash. Added blockImagesAndStyles (default on).
Round 2 - stop rotating on every 403. Session reuse exposed the next problem: Upwork rate-limits individual job detail URLs while the same session serves search pages fine, and treating that as a burned identity meant a browser restart plus a fresh Cloudflare solve per retry - 3.7 minutes burned on one row that was skipped anyway. Rotation policy is now per call site: search pages rotate on a block (the query is unreachable otherwise), detail pages take one shot and fall back to the card-level row. A genuinely burned identity is still caught by a consecutive-block counter.
Round 3 - don't confuse "don't rotate" with "don't retry". Round 2 stopped retrying detail pages
altogether, which held cost down but gutted the data: a 20-item platform run came back with 10 of 20
rows card-only. Upwork's per-URL limit is transient, and a same-session retry costs one page load
rather than a browser restart plus a challenge solve. New detailAttempts input (default 3) sets
that budget. Solving is also no longer forced merely because attempt > 0 - that was right when a
retry implied a new session, and wasteful once retries stay on the same one.
RUN_STATS gained sessionsStarted, blockedFetches, detailFetchFailures and cloudflareCleared.
sessionsStarted: 1 is the health signal.
Measured on the Apify platform, residential proxy, 20 items with detail scraping:
| Build | Results | Time | Cost | Per result | Rows with client block |
|---|---|---|---|---|---|
0.1.3 browser per page | 5 | 9m14s | $0.557 | $0.1114 | - |
0.1.4 session reuse | 11 | 10m58s | $0.453 | $0.0412 | - |
0.1.5 rotate only when burned | 20 | 8m9s | $0.336 | $0.0168 | 10/20 |
0.1.6 + same-session retries | 20 | 10m1s | $0.329 | $0.01645 | 19/20 |
Cloudflare was solved twice across 24 fetches in the 0.1.5 run, against once per page before.
client.totalSpentUsd is present on 7/20 - that is the source's truth, not a gap: the other 13
clients are new accounts with no spend history.
Where the money actually goes (apify runs info <id> --verbose, per-line, not derived):
| Build | Compute | Residential bytes | Residential $ | Total |
|---|---|---|---|---|
0.1.3 (4096 MB) | 0.6156 CU / $0.246 | 33.10 MB | $0.308 | $0.557 |
0.1.4 | 0.3656 CU / $0.146 | 32.60 MB | $0.304 | $0.453 |
0.1.5 | 0.2717 CU / $0.109 | 24.22 MB | $0.226 | $0.336 |
0.1.6 | 0.3338 CU / $0.134 | 20.74 MB | $0.193 | $0.329 |
Residential proxy traffic is 59% of the bill on 0.1.6; compute is 41%. That is the single most
important number for cost work: wall-clock optimisation only attacks the smaller share. Roughly
1.06 MB of billed traffic per fetch, after blockImagesAndStyles has already dropped images,
fonts, media and CSS.
Rates this account is billed at, consistent across all four runs: $0.40/CU and $9.31/GB residential. Both are above Apify's published list rates ($0.20/CU, $8/GB) - worth checking which plan is active.
Peak memory never exceeded 764 MB in any run, including 0.1.3 which was allocated 4096 MB.
Tests 126 → 146.
Second platform run scraped fine and then failed on push with
InvalidRequestError: Schema validation failed. Apify validates every pushed item against the
fields block of .actor/dataset_schema.json, and the fields were declared as bare
"type": "integer" / "string". The actor deliberately emits null for anything Upwork did not
show - design rule 3, "None never becomes 0" - and JSON Schema "type": "integer" rejects null. The
null discipline broke its own schema.
client field) is now ["<type>", "null"].tests/test_dataset_schema.py, five tests guarding the contract: all fields nullable, no
required list, no field emitted-but-undocumented, no field documented-but-never-emitted, and no
view referencing an unknown field.category,
client.rating and client.companyName, which had been deleted from the schema because they do
not exist logged out. Removed from the parser too, so no always-null columns reach a buyer.Tests 121 → 126.
First run on the Apify platform succeeded with 0 results: every fetch died instantly with
BrowserType.launch_persistent_context: Executable doesn't exist at /pw-browsers/chromium-1194/chrome-linux/chromeRoot cause: scrapling's StealthyFetcher drives Chromium through patchright, a patched
playwright fork (scrapling/engines/_browsers/_stealth.py imports patchright.sync_api and calls
playwright.chromium.launch_persistent_context). It has nothing to do with Camoufox - scrapling
never imports camoufox at all. The Dockerfile was downloading ~200 MB of Camoufox for nothing while
the browser actually needed was absent.
It passed locally only because a matching Chromium revision was already sitting in
~/.cache/ms-playwright from an unrelated project, which masked the missing install right up until
the actor ran in a clean container.
python -m patchright install chromium instead of camoufox fetch, and lists
the resulting browser directory so a future regression is visible in the build log.camoufox[geoip] dependency. The browser binary is not a pip package and cannot live
in requirements.txt.Moved out of the claude-code prototyping monorepo into a standalone repo, ahead of publishing to
the Apify Store. The monorepo commits real PII to main and is a scratchpad for one-off tasks; this
is a product with its own release cycle.
The _vendor-tqm/ snapshot of the upstream lead-gen scraper stays behind in claude-code - it
contains TQM's commercial ICP logic and has no place in a repo backing a Store listing. See
docs/PROVENANCE.md.
Added docs/: ARCHITECTURE, DEVELOPMENT,
UPWORK-DOM, PROVENANCE.
First live run returned zero usable rows on a 104/104 green test suite. Five real bugs, all fixed, all now covered by tests; full detail in docs/UPWORK-DOM.md.
data-test on Upwork is space-separated and multi-valued, so every = selector silently matched
nothing. Switched to ~=.<span class="highlight"> wrappers; text is now joined
across descendants instead of taking the first node.$3.00/hr as the client's $3 lifetime spend. The
bare-dollar fallback is gone; client regexes are scoped to the client section.hiringOrganization is literally {"name": "Upwork"} - was populating
companyName.Product-level finding: logged-out search cards carry no client data at all. The client block now
requires scrapeJobDetails: true. Knock-on changes:
client.paymentVerified is tri-state (True/False/None). Returning False for "not shown"
made paymentVerifiedOnly reject all 37 jobs in the first run.payment_verification_unknown / client_spend_unknown separately from
*_below_min, so a zero-row run points at scrapeJobDetails instead of leaving buyers guessing.Also in this pass:
data-* hooks and is parsed as
li.ca-item title/value pairs: interviewingCount, invitesSent, unansweredInvites,
lastViewedByClient. New competition dataset view to go with them.activeHires, totalHours, industry, companySize.<strong> amount with its sibling
.description label. Upwork renders a bare Fixed price with no number on most search cards.jobType is read from the label independently of the amount, so it populates even when no budget
is stated.payment_verified=1 is no longer sent to Upwork - its 307 redirect strips the param, producing an
unfiltered result set that looked filtered. t= and contractor_tier= were each tested against
live results and confirmed to work.apify pinned to >=4.0.0: 2.x/3.x ship a crawlee that crashes on import against the
browserforge that scrapling[fetchers] requires.Generalized from the TQM Invest in-house Upwork scraper into a standalone Apify actor: dataset
output instead of a Postgres insert, arbitrary search queries instead of three hard-coded ones,
every filter optional and caller-parameterised, proxy rotation with retry-on-block, cross-query
dedup on jobId, and server-side narrowing via search-URL params.
src/ (normalize, filters, urls, parser, fetcher, main), .actor/ manifests with a ~30-field input
schema and a fully documented dataset schema, Dockerfile on apify/actor-python-playwright:3.12,
and 104 unit tests.
failOnZeroResults told buyers: "charges are per result, and a run with no results incurs none."
That stopped being true the moment monetization went live, because Apify's price form pre-fills an
apify-actor-start event and it was saved untouched. A zero-result run then costs $0.00005.
The money was never the point — a buyer running hourly for a month pays about 3.6 cents in start fees. A written promise in the listing was simply false, and nothing would have caught it: the claim lives in an input description, the charge lives in Console, and no build, test or run compares the two.
"Leave the default" is not a neutral act on a pricing form. The default was not wrong; it just was not a decision, and it silently contradicted one that had been made.
Sentence corrected to state the start charge explicitly. The event is being kept, at the $0.00001 minimum rather than removed — see the decision below.
Removing it forfeits a waiver Apify states outright in its removal dialog: "if you remove this event, the first 5 seconds of run costs won't be waived." That waiver lands on our platform bill, not the buyer's.
Costed against a compute model validated on real runs (it reproduces a measured
ACTOR_COMPUTE_UNITS of $0.07092 as $0.07089):
| Waiver worth to us | $0.00056 – $0.00222 per run, at 1–4 GB |
| Start fee cost to the buyer | $0.00001 per run |
| Ratio | 56× to 222× in our favour |
And it is far larger than that on short runs: a single-HTTP-request actor finishing in ~6 seconds has 83% of its compute waived.
Deleting the event would mean paying roughly $0.0011 to save a buyer $0.00001. 0 is not accepted —
the form enforces a $0.00001 minimum — so the choice was only ever "minimum" or "none".