- Removed the missing-next-link auto-wait, skipped unused stylesheets, and gated extraction on actual page readiness.
- Bounded queued enrichment by the live charge budget and deduplicated scheduled detail work.
- Added failure circuit breaking and performance/transfer diagnostics; retained the measured 1 GB browser allocation.
- Report confirmed custom charge counts and reject invalid URLs before crawling.
Second repair pass on the same Apify automated-QA failure. The Actor was still flagged Under maintenance
because the daily Store test only ever runs the build tagged latest, and latest was still 1.0.28 from
2026-10-14 — the build with the three-attempt single-pool budget that the 2026-10-19 work replaced. The repaired
code sat on the qa-reliability tag, where Apify never tested it, so the flag could not clear no matter how well
that build performed.
-
Promoted the repaired build to latest, which is the change that actually exposes the fix to the daily test.
-
Raised the first-contact cap in src/limits.ts from 170 s to 220 s. Nine runs of build 1.0.41 against the
exact input-schema prefill gave 8 passes and one failure, fSiKnrLMfi3hIxI7J: it gave up at 174 s after 7
challenges while the ladder was still handing it fresh exits, with roughly 130 s of the five-minute window
unused. 170 s spent 57% of the window; 220 s spends 73% and still leaves about 45 s of margin once cold
container boot, the extract/push/charge tail, and the 5 s guard interval are counted. test/limits.test.ts
now asserts both halves of that bound so neither can drift.
-
Stopped reporting an untouched request queue as an empty result. Apify's RequestQueue is eventually consistent:
the seed added moments earlier can be missing from the queue head, and Crawlee then shuts down having processed
nothing. Measured during this repair's own matrix — run B9viVhlueMM3OjlyH, the exact QA prefill, 0 requests in
2.5 s — which the old code announced as "zero matching public records". A run that never fetched anything has
learned nothing about Dubizzle's inventory, and Apify's daily test reads a zero-row success as a failure anyway.
The crawler now re-seeds once with fresh unique keys, and classifyRunOutcome in src/limits.ts separates
delivered / blocked / queue-fault / empty so the run fails honestly instead of claiming an empty result.
The rest of the 2026-10-19 reliability work is unchanged: the cost-ordered proxy ladder, the one-challenge budget
for the metered Apify Residential pool, the 12 s first-contact readiness poll, and transport-failure separation.
Repair for the Apify automated-QA failure that flagged the Actor Under maintenance. The daily Store test runs
the Actor with its input-schema prefill values and requires a succeeded run with a non-empty dataset within five
minutes. Build 1.0.28 allowed three attempts in about sixty seconds on a single proxy pool and then called
Actor.fail, so one challenged pool failed the test while four minutes of the window went unused.
Reliability
- Cost-ordered proxy ladder in
src/proxy.ts (resolveTieredProxy, policy in createLadder): DataImpulse
residential, then DataImpulse mobile, then Apify Residential, then back to the cheapest. A pool is left only
after it has actually been challenged, so the cheap pool stays preferred.
- First-contact budget (
firstContactBudgetMs / shouldStopFirstContact in src/limits.ts): until a record has
been delivered, retries continue across pools for 75% of the run budget capped at 170 s. maxRequestRetries
2 -> 20 and maxSessionRotations 3 -> 25, so elapsed time rather than an attempt count is what ends first
contact. Once a record is delivered, errorHandler restores the previous three-attempt policy and large runs
stay cheap.
- The budget is keyed on delivered records, not on pages read, and the run-level guard enforces it directly. A
page can pass the anti-bot checks, bump the page counter and then fail during extraction; keying on pages
switched the budget off, and because a Store-triggered run carries the Actor's one-hour platform timeout,
nothing else bounded it — two measured runs drifted to 271 s and 279 s, missing the five-minute window.
- Transport failures (
ERR_TUNNEL_CONNECTION_FAILED and friends) are separated from genuine challenges. A dead
tunnel buys a fresh session on the same pool instead of consuming the escalation budget.
- Readiness poll is 12 s during first contact, down from 25 s: a readable page settles in about six seconds.
It must not go lower. Build
1.0.37 tried 7 s from the third attempt on to buy more exits per second; it
bought them and they were worthless, with failing runs going from at most 6 challenges to 11-13 because
Imperva's interstitial needs roughly that long to solve itself. Reverted, and recorded so it is not retried.
Platform-usage cost, from measured usage rather than assumption
- Across a 24-run cloud matrix, Apify Residential transfer at $8/GB was $0.014-$0.024 per challenged run and
51-64% of that run's entire platform cost, while a 181-second run that never touched it cost $0.0165 in total.
Long runtime is affordable; the metered pool is not.
- Apify Residential is therefore
meteredByApify and worth exactly one challenge and one visit per run, after
which the ladder cycles only the externally billed DataImpulse pools. Measured effect across the matrix:
residential transfer $0.0937 -> $0.0515, a 45% reduction.
- Lowering the first-contact cap to 170 s bounds a hopeless run at roughly 185-200 s of wall clock instead of
271-279 s, cutting its compute accordingly.
- On the shape Apify tests daily the recurring cost is unchanged: mean platform usage per run $0.0034 on build
1.0.28 against $0.0037 on the repaired build. The extra spend falls only on runs that previously failed in
sixty seconds and delivered nothing.
Diagnostics and packaging
OUTPUT reports proxyTier, proxyLadder and proxyVisits, so a run's cost can be attributed to a pool
after the fact, and the blocked-run terminal message names the pools tried.- Added
.actorignore so local reports and scratch directories are no longer uploaded with the Actor source.
- Enforced clean alphanumeric sticky session generation in
formatDataImpulseUrl to eliminate initial ERR_TUNNEL_CONNECTION_FAILED retries.
- Implemented bounded detail queue scheduling in
src/main.ts: only queues detail requests up to maxResults instead of flooding Request Queue with entire search pages, slashing queue writes by >90%.
- Accelerated
waitForUsablePage challenge polling from 800ms to 250ms and removed redundant domcontentloaded wait, cutting page detection time by ~70%.
- Shortened
humanizePage delay from 1.8s to 200-500ms to save container compute seconds on every request.
- Fixed response listener memory buildup by clearing
response listeners per request and freeing capturedResponses immediately after parsing.
- Pinned
autoscaledPoolOptions concurrency to prevent Crawlee's Snapshotter from throttling and stalling execution when memory nears limits.
- Expanded network request aborting to block 40+ third-party telemetry, A/B testing, and advertising tracker domains.
- Increased
maxRequestsPerMinute to 60 (from 18) to remove artificial 3.3s delays between requests.
- Integrated DataImpulse residential proxy support with sticky sessions and automatic fallback to Apify Residential proxy.
- Mapped
@DATAIMPULSE_PROXY_URL, @DATAIMPULSE_ACTOR_PROXY_URL, and @DATAIMPULSE_MOBILE_PROXY_URL in .actor/actor.json.
- Eliminates Apify's $8.00/GB residential proxy bandwidth transfer charge during cloud runs, reducing platform usage costs by up to 90%+.
- Enforced
defaultMemoryMbytes: 1024 and capped maxMemoryMbytes: 2048 in .actor/actor.json to eliminate oversized 4096 MB allocation and slash compute unit consumption by 75%.
- Bounded
maxRequestRetries to 2 (from 5) and maxSessionRotations to 3 (from 5) in src/main.ts to prevent runaway retries from burning residential proxy bandwidth and compute during persistent Imperva challenge blocks.
- Re-aligned
detail-result list price to $0.036 (up from $0.012) across .actor/pay_per_event.json, src/main.ts, schema, and documentation to cover measured Playwright + residential proxy cost ($0.012 - $0.018/page) and secure a positive 39% - 51% net operating margin across all pricing tiers.
- Switched proxy back to native Apify Residential proxy (
groups: ['RESIDENTIAL']).
- Optimized compute power and bandwidth: blocked heavy images, media, fonts, and third-party advertising/tracking scripts at network level while preserving image URL extraction from HTML DOM attributes.
- Eliminated
slowMo: 75 delay and added lean Chromium flags (--no-sandbox, --disable-dev-shm-usage, --disable-accelerated-2d-canvas, --disable-gpu) for faster, lightweight page rendering.
- Reduced default container memory to 1024 MB in
actor.json, cutting compute unit consumption by 50%.
- Configured
retryOnBlocked: false and sessionPoolOptions.blockedStatusCodes: [] on PlaywrightCrawler to prevent Crawlee from preemptively aborting on Imperva HTTP 403 challenge pages before requestHandler can execute and solve.
- Resolved Imperva challenge handling: added challenge-resolution polling in
waitForUsablePage allowing Imperva client scripts to solve, set verification cookies, and reload naturally before block checks run.
- Refactored
looksBlocked and isBlankPage checks to prevent premature session rotation while challenges are solving.
- Fixed detail DOM extraction by querying selectors directly without nested closures, preventing runtime evaluation errors.
- Synced
.actor/pay_per_event.json and documentation with active Store tiered event pricing.
- Broadened the residential proxy pool instead of pinning every retry to UAE exits. Three separate Apify QA incidents showed the UAE-only pool returning Imperva challenge pages for every session despite the one-result default.
- Reduced the public health-test prefill and defaults to one property result and one page. Larger user runs remain opt-in through the input limits; this limits exposure to Dubizzle's intermittent Imperva challenge during Apify's daily default-input test.
- Added Dubizzle property and car search mode with URL and structured-search inputs.
- Added direct listing detail extraction and optional search-result enrichment.
- Added property broker directory and broker-profile extraction.
- Added UAE residential sessions, Imperva challenge detection, retry controls, typed schemas, run summaries, and PPE cost caps.
