Fixes every search returning 0 businesses since Yelp put its pages behind
DataDome. yelp.ie used to be unprotected; it now answers every HTML request with
DataDome's "enable JS" interstitial as HTTP 403 — from six different RESIDENTIAL
IPs in one run. Rotating residential IPs cannot clear that wall.
- Escalate a 403 to Apify Unblocker. A 403 is a bot wall, not a throttled IP:
the page is retried once through the
UNBLOCKER proxy group, which solves the
challenge, instead of spending the retry budget on more IPs the wall refuses.
Unblocker is billed per successful request and is never touched unless a wall
is actually hit, so unprotected runs are unchanged in cost.
- A challenge page served as HTTP 200 is rescued too. DataDome can return its
interstitial with a 200; those short captcha pages are now recognised and
escalated the same way, rather than being reported as "no businesses".
- Every Unblocker attempt is budgeted by the run's own deadline. A walled
page used to get two full 150s attempts regardless of how much of the window
was left; on a 300s run four attempts across two search pages consumed the
entire window (run
PghocZ07GTe4b1Hzc, build 1.0.73). Each attempt now gets
min(150s, time left - 12s), and no attempt starts with under 20s left, so a
failing page can no longer starve the pages after it.
- A run stopped by its own deadline with nothing extracted is a block, not a
success. The block check ran only on normal completion, so the run above
ended
SUCCEEDED with "Stopped at run timeout with 0 businesses" — exactly the
silent failure this actor was fixed to report.
- Once a page is walled, a failed rescue is retried. After one Unblocker
attempt died, the follow-up residential attempts failed with timeouts instead of
another 403 and the page was lost unreleased (run
QVRtW1ymascxpkr5f, build
1.0.72). Seeing a wall once now marks the URL as walled, so any later failure on
it goes back to the rescue (still capped at two attempts) instead of burning a
rotation on an IP the wall would refuse anyway.
- Search results now come from GraphQL too, so a normal run never touches the
wall. The result list used to exist only in the
/search HTML page's Apollo
cache; Yelp's /gql/batch answers the same question in ~200ms and is not behind
DataDome. One persisted query returns up to 25 businesses (hard server cap, no
cursor), after which the walled HTML pages are still used for the remainder —
which is why the Unblocker path stays. Results, cost and speed for the common
maxResults <= 25 run improve on both the pre-DataDome and the walled versions.
- Business details now come from GraphQL, not a walled HTML page. Yelp's
/gql/batch is not behind DataDome, so phone, website, amenities, about and the
map link come from one ~200ms persisted query instead of a /biz/... page that
cost an Unblocker round trip (22-67s, $0.0015) per business. A rotated Yelp
documentId degrades to the HTML page, never to a business with empty fields.
This makes every run — walled or not — faster and cheaper than before, since the
per-business HTML page is no longer fetched at all.
- The rescue hop gets its own timeout. Solving DataDome takes 70-100s, so the
rescue attempt runs on a 150s budget while ordinary attempts keep the 12s cap;
unwalled runs are not slowed at all.
- The run still fails visibly if the rescue fails — a fully walled run is a
failure, never a silent empty result.
Fixes a run that was killed at the platform timeout (TIMED_OUT) after going
idle mid-page. It had already written 16 good businesses; nothing stopped work
before the kill, so the whole run was reported as a failure.
- Never hit the platform timeout again. The actor now derives a deadline from
the run's own
timeoutAt, stops starting new work once it passes, and finishes
cleanly with whatever it collected — a SUCCEEDED run with partial results
instead of a TIMED_OUT failure. The margin scales with the window (10%,
clamped to 5-20s) so short QA validation runs are not eaten by a fixed reserve.
- A stalled listing can no longer hold the pipeline. Each business is bounded
by its own time budget; one that outlives it is abandoned with a warning rather
than holding all 15 concurrent workers open. This was the actual shape of the
failure — 4m37s of silence at 0.9% CPU, waiting on a single await that never
settled. (HTTP fetches were already bounded at 12s/attempt; the stall was in a
non-HTTP await, where the Apify client retries silently for minutes.)
- Watchdog backstop. If something hangs where no loop can observe the
deadline, a timer finalizes the run and exits 0 before the platform kills it.
- Fixed a proxy socket leak. Every request built a
ProxyAgent that was never
closed; undici holds keep-alive connections until an agent is closed, so each
request leaked one TCP connection to proxy.apify.com for the life of the run
(measured: 20 requests -> 20 sockets still open, 0 after the fix). A fresh agent
per request is kept deliberately — the residential proxy assigns an exit IP per
connection, so pooling would pin every rotated retry to the same IP.
- Bounded review pagination. Reviews with empty text are skipped, so the
"collect N reviews" loop could page forever on a business with thousands of
them. It now stops a few pages past the ideal count.
- Fix
maxResults to apply per search term/location pair (matches INPUT_SCHEMA and README); global dedup unchanged.
- Prevent concurrent workers from exceeding per-search
maxResults via slot reservation (run-scheduling.ts).
- Parallelize business detail page + review GraphQL fetches when both are needed (lower wall-clock on review cohorts).
- Parallelize independent search term×location pipelines (up to 3 concurrent) for multi-city inputs.