Verified, did not re-derive: 0.1.0's run-budget + bounded-concurrency
fix (below) was already on main, already pushed as build 0.1.1 on
Apify, and already regression-tested (
,
, ). Re-ran the full suite (76 passed),
ruff and pyright clean; nothing there needed changing.
Found while re-checking .actor/actor.json: defaultMemoryMbytes
was 4096 but maxMemoryMbytes was 2048 — a default above the
configurable ceiling, left behind by the 2026-09-19 "every QA run
measures a memory config no customer runs" finding, which raised the
default to match the platform's real 4096 MB but never touched the
range around it. Raised maxMemoryMbytes to 4096 so the manifest is
internally consistent again; would likely have broken (or silently
clamped) the next apify push.
Cloud evidence checked, not re-run: the last cloud run on this
build (0Gx63vg1u2JEq8vBz, 2026-09-19T09:27) FAILED — but on a seed-level
403 exhausting all 5 retries (SeedBlockedError, REQ-7's fail-loud path
working as designed), not a timeout and not a code defect. No cloud run
has yet measured the fixed code's actual wall-clock/cost at the shipped
prefill; that is the orchestrator's cloud-QA step to run next.
Root cause (measured on Apify the same day): every fetch — seed,
product-type listing page, AND every individual product-detail page —
was paced PAGE_DELAY_S=15s apart serially, one at a time, with no
concurrency. At full prefill depth (3 seeds x up to 50 items each =
up to ~159 total fetches), that's ~40 minutes of pure pacing delay
alone, before any real HTTP latency. Four cloud runs the same day:
0 rows/TIMED-OUT, 9 rows/SUCCEEDED, 75 rows/TIMED-OUT at exactly 30
min, 100 rows still RUNNING 37+ min in — not crashing, just far
slower than any sane run budget.
Fix 1 — bounded concurrency: DETAIL_CONCURRENCY=3 on
product-detail fetches only (the independent, per-item class of
request) via a small pool of independent DirectIndustrySession
instances. Category/product-type navigation fetches stay
sequential/paced — the fewer, WAF-sensitive discovery steps the live
probe actually proved safe at the 15s cadence; each concurrent detail
batch still shares one PAGE_DELAY_S pause, so the per-15s-window
request rate triples rather than going unbounded.
Fix 2 — run-wide wall-clock budget (src/run_budget.py, same
shape as cars-com-listings-scraper's 0.8.0 /
's 0.7.1 fix): derives the crawl's deadline from the REAL
platform kill clock (Actor.configuration.timeout_at), not a guessed
constant. Threaded through every crawl-loop boundary; once the
deadline passes, the walk stops at the next checkpoint and the run
ends SUCCEEDED with whatever rows it already has — never
TIMED-OUT. RunStats.budget_exceeded drives an honest partial-result
status message instead of a silent "Done".
Expected wall-clock at full prefill depth (3 seeds, 50 items/seed, 5
pages/product-type): ~15 minutes, comfortably inside both the 20-minute
local fallback budget and the platform's real deadline minus its
300s safety margin.
![ThomasNet Scraper [$0.9/1K💰] | Supplier Data | B2B Leads avatar](https://images.apifyusercontent.com/u1AjLQtlv9qQokNMt08CCc9RsIpLzx32SGOWnqe4eQs/rs:fill:76:76/cb:1/aHR0cHM6Ly9hcGlmeS1pbWFnZS11cGxvYWRzLXByb2QuczMudXMtZWFzdC0xLmFtYXpvbmF3cy5jb20vZHRSN3FuVGJRck5pU0NtajctYWN0b3ItZEVuM0NucXpQMG03SzFGMzItVXk3QTdLU1hLdC1pbWFnZXNfJTI4NCUyOS5qZmlm.webp){kind=avatar}
