Turn the Clutch.co agency directory into B2B leads for lead generation: 19 always-present company data fields, public client reviews, the agency's real website, and an optional AI ICP-fit score. $2.50/1,000, 98.1% complete. Never charged for an empty or blocked result. JSON/CSV, API, schedule.
All notable changes to the Clutch.co B2B Agency Intelligence actor are documented here.
This project adheres to Semantic Versioning .
Version note. Two numbers run in parallel and they are NOT the same: this file and
package.json track the SOURCE version (0.2.x); Apify tracks the deployed BUILD under actor
version 0.1 (0.1.4, 0.1.5, …, tag latest). Each section below names both.
[0.3.0] — a headful browser earns the pass, HTTP keeps the scrape (outage since 2026-08-06)
Outage window: the last delivering run finished 2026-08-06 10:28Z. Every run since has
returned nothing — clutch.co began answering got-scraping with HTTP 403,
cf-mitigated: challenge
and a "Performing security verification" interstitial on the very first fetch, from
every proxy tier. Attribution is target-side, not ours: two different builds failed identically
in the same minute, a forced-datacenter control failed too, and plain curl from the owner's own
Mac failed as well. No HTTP client can pass a JS challenge — it has to be solved by a browser.
This release RESETS the Gate #1 clean-day streak — it changes the fetch layer.
The measurement that shaped the fix (owner's Mac, one IP, ~40 minutes, 2026-08-08)
handoff to got-scraping, generated headers left unpinned
⚠ 1 of 4 pages — a coin flip
handoff with headers pinned to the solving browser, after cf_clearance appeared
✅ 5 of 5, HTTP 200, 0.86–3.1 MB each
Two facts decided the design: the browser must be headful (headless failed every attempt,
persistent profile made no difference), and the Cloudflare pass transfers — once
cf_clearance is issued, plain got-scraping can keep fetching from that exit IP, provided the
request carries the browser's cookies, its exact User-Agent, and a headerGeneratorOptions
pinned to the same Chrome major version. The 1-of-4 failure was a self-contradicting fingerprint
(overriding only the UA while got-scraping generated a fresh header set per request); the run
that scored 5/5 is the one whose cookie jar held cf_clearance, not just __cf_bm.
Added
src/fetch/clearance.js — a challenge solver + TTL cache. A headful, stealth-launched
Chrome opens once per proxy session, blocks image/media/font/stylesheet requests (the solve
needs neither and every byte is billed residential bandwidth), polls until the page stops
looking like a challenge AND cf_clearance is present, then hands back the cookie header, UA
and a pinned headerGeneratorOptions. Cached per session for 25 minutes (CLEARANCE_TTL_MS);
concurrent callers share one in-flight solve; a solve that exhausts its retries degrades to
unauthenticated rather than crashing the run, so a total block still fails honestly through the
existing blockedOut guard instead of a stack trace.
src/fetch/transport.js wires it in as an optional collaborator: clearance: null keeps
today's behaviour byte-for-byte. When present, each proxied request first ensures a pass on the
same exit IP the sticky session just minted, merges the pinned cookie/UA/header-generator set
into the got-scraping call, and invalidates the clearance the moment a session is burned — a
solve is worthless on a different exit and must not outlive the IP that earned it.
transport.stats gains solves, solveFailures, clearanceReuses so the probe can tell "one
solve carried 500 pages" from "we re-solve every other request", the cost question that matters
before the bill arrives.
src/main.js dynamically imports playwright inside main() only, inside a try/catch
— no browser available, no crash, just a warning and the old HTTP-only path. New env knobs:
SOLVER_ENABLED (default true), SOLVER_HEADLESS (default false — headless was
measured to fail), SOLVER_TIMEOUT_MS (90 000), SOLVER_MAX_ATTEMPTS (2),
SOLVER_FAILURE_COOLDOWN_MS (120 000), SOLVER_MAX_WASTED_SOLVES (2),
CLEARANCE_TTL_MS (1 500 000), SOLVER_BROWSER_CHANNEL
(default empty ⇒ Playwright's bundled Chromium; the live measurement used channel: 'chrome',
and which one the Apify image can pass with is a harvest question, hence a knob and not a
constant). SOLVER_URL is read and reported but reserved: the solve always targets the URL
of the request that triggered it, so overriding it only matters if a future warm-up solve is
added. Run OUTPUT now carries solves/solveFailures/clearanceReuses next to proxySessions.
A failure cooldown on the store. An exhausted solve is remembered for
SOLVER_FAILURE_COOLDOWN_MS (120 s) and every request inside that window degrades immediately
instead of launching browsers again. Without it a fully-walled-off target costs
SOLVER_MAX_ATTEMPTS × SOLVER_TIMEOUT_MS (up to 180 s) per request — one such request eats
the entire 300 s auto-test wall on its own. A successful solve clears the cooldown.
A wasted-solve cap on the store. The failure cooldown above only remembers solves that
failed; the more expensive runaway is a pass that is earned and then refused anyway. Every
detected block invalidates the clearance and FetchClient retries a blocked URL up to
MAX_BLOCK_RETRIES times, so each retry would mint a new exit IP and launch a fresh headful
Chrome — minutes of wall clock that BLOCK_BACKOFF_PER_REQUEST_MS does not bound (it caps
sleeping, not solving) and RUN_DEADLINE_MS cannot interrupt (it is only read between
targets). After SOLVER_MAX_WASTED_SOLVES (2) passes in a row that no later request ever
reused, the store degrades to the cheap unauthenticated path for one cooldown and says so in
the log. A pass that carried even one further request resets the counter, so healthy runs never
meet this.
Two timeout tiers, not one.FetchClient's abort window is
30 s + SOLVER_TIMEOUT_MS × SOLVER_MAX_ATTEMPTS, because the solve runs inside the request it
is unblocking; at the old fixed 30 s the first proxied request aborted mid-solve every time.
That window applies to every request, though, and only ~one per proxy session actually
solves — so the transport now arms its own 30 s leash (httpTimeoutMs) around the
got-scraping call alone, chained to FetchClient's signal. Without it a hung residential exit
would inherit the whole 210 s window per attempt, and ~420 s before dead-exit rotation abandons
it: a straight regression of the 2026-08-05 incident (run Yw5MYnZBUAVsdhXHc) that rotation was
written for, against a 180 s run deadline and the 300 s auto-test wall.
The solve deadline covers the whole attempt.solveTimeoutMs is armed before the browser
launch and is passed to page.goto, instead of bounding only the poll loop and letting
Playwright's independent 30 s navigation default sit outside it. Otherwise one attempt could run
launch + 30 s + the full poll budget ≈ 125 s, two attempts ≈ 250 s inside the 210 s window
above — killed mid-solve by the very window that exists to prevent that.
A poll that lands mid-navigation no longer kills the attempt. Cloudflare's interstitial
reloads itself while it works, and page.content() in that window rejects with "Execution
context was destroyed" — the challenge working, not a defect. It is now swallowed and re-polled
(empty content still reads as "not cleared", so nothing passes on stale state); if it never
recovers, the timeout message names the read error.
The fingerprint pin now covers the operating system too.buildHeaderGeneratorOptions
derives operatingSystems from the same user-agent it pins the Chrome major off, instead of
leaving the key unset at the only call site. Unset is not neutral: got-scraping samples an OS
freely, so the generated sec-ch-ua-platform could say macOS on a request whose User-Agent we
have force-pinned to the solving browser's Linux one — the 1-of-4 self-contradiction, moved from
the browser version to the platform headers. An unrecognised UA drops the key rather than
guessing.
Changed — base image, dependency, memory
Dockerfile base swapped apify/actor-node:20 → apify/actor-node-playwright-chrome:20 —
a headful browser needs Playwright's browser binaries and headful OS libraries preinstalled,
which the plain Node image does not carry. Manifest-first layer caching kept. Two consequences
of that image, both required: COPY --chown=myuser:myuser (it runs as non-root, so root-owned
files break npm install with EACCES) and CMD ./start_xvfb_and_run_cmd.sh && npm start (a
headful browser cannot start without a $DISPLAY). The <node>-<playwright> tag and the exact
playwright version must be pinned together at harvest — the image ships the browser and
skips its download, so a mismatched pair fails with "Executable doesn't exist at …".
package.json: playwright added as a dependency (the one new runtime dependency this
release is approved for); version 0.2.11 → 0.3.0 — a base-image and fetch-architecture
change is not a patch.
.actor/actor.json: minMemoryMbytes: 2048. A headful Chrome under the platform's virtual
display does not fit the old footprint, and the daily Store auto-test runs on the actor's
default memory — an OOM there is three consecutive failures away from an "under maintenance"
badge. ⚠ This floor does not reach the three live schedules, which still carry their own
memoryMbytes: 1024 and must be PATCHed to ≥2048 in the same deploy (harvest step).
Unchanged
Pricing. The cheap HTTP scrape path is untouched; the browser exists only to earn the pass, once
per proxy session, instead of the ~$2.04/1000 profiles a browser-fetches-everything design would
cost against $2.00/1000 of kept revenue. ⚠ The new cost is not the old $0.719/1000: a solve
adds ~2 GB of memory for the whole run plus browser wall-clock and its own residential bandwidth.
How much it adds is unmeasurable offline and is the first number the restarted Gate #1 probe has
to produce — treat the old figure as a floor, not a forecast.
Every existing transport test, unchanged and passing with clearance absent.
Post-completion (harvest, outside this build)
Real Cloudflare, real proxy, on-platform headful check, base-image swap verification, the image
tag + playwright version pin with a regenerated package-lock.json, raising the three live
schedules to ≥2048 MB, re-deriving RUN_DEADLINE_MS/SOLVER_TIMEOUT_MS/SOLVER_MAX_ATTEMPTS
against a measured {} run, and the Gate #1 probe restart are done live at harvest, not inside
this offline change.
[0.2.11] — the daily health check can no longer be failed by a slow day (2026-08-06)
Apify runs every Store actor once a day on its default input and requires SUCCEEDED with a
non-empty dataset inside 5 minutes; miss it twice in three days and the public page gets an
"under maintenance" badge — direct damage to discovery, which is this actor's biggest business
risk. Two live runs measured 364 s and 306 s: both delivered, both were honest, both would
have failed that check. The cause is arithmetic, not a defect — the block-backoff budget is 240 s
wide and per RUN, so a bad day on the target can spend it on a run of any size.
Two independent guards, because either alone leaves half the problem:
Changed
Default run size 25 → 5 agencies (editor prefill 25 → 10). This shrinks the fetching half
of a slow run. It is also what the rest of the Store does — the published guidance among Apify
developers is a 5–10 default, and the largest rival Clutch actor ships 20 on a cheaper path that
does no listing crawl. A first run now costs a buyer $0.0125 instead of $0.0625; anyone doing
real work raises the number, and nothing caps it.
Added
A wall-clock wind-down for small runs. At 180 s a run stops starting new work, finishes the
agency already in flight, and exits cleanly with what it has. Bounded by design: the deadline is
only consulted between agencies, and the one in flight can still absorb the full 60 s per-request
block ceiling, so the true ceiling is ~180 + 60 + fetch + container boot ≈ 265 s — about 35 s
of margin under the 300 s wall. Tunable via RUN_DEADLINE_MS.
Scoped, so no buyer is ever truncated by a clock. The deadline is armed only for a run of
≤ 10 agencies (RUN_DEADLINE_MAX_AGENCIES) — the daily auto-test and a first-click trial. A
500-agency harvest gets no deadline at all and runs for however long 500 agencies take. A test
pins the coupling: raise the schema default above that number and the suite goes red rather than
silently disarming the guard.
windDown / windDownReason in the run report, kept separate from degraded and
chargeLimitReached. Three different stop reasons that must never be confused: the target
walling us off, the buyer's budget running out, and us choosing to stop early. Gate #1 reads
degraded as a reliability verdict, so folding a clock stop into it would report a healthy
actor as blocked.
A wind-down that delivered nothing now FAILS the run. Winding down is billing-neutral and
normally partial-but-honest, so it exits SUCCEEDED — but with zero rows it is indistinguishable
from a healthy empty run, and the existing blockedOut guard does not catch it (that needs
every fetch blocked, or a degrade). Closing that hole in the same release that opens it: the
wind-down must not become a new way to lose data silently.
Billing is untouched: everything delivered is still charged, everything not delivered is still
free. The wind-down verdict is not restored on resume — the deadline is measured from the
segment's own start, so a resumed run gets a fresh budget.
[0.2.10] — the changelog link was dead, and the page claimed one release ever (2026-08-06)
Docs-only; no runtime code changed. Owner decision: a visible, maintained changelog is a trust
signal worth carrying on the Store page — a visitor who can see the product improving is more
likely to try it. So the section that carries that signal has to actually work.
Fixed
The Changelog link on the Store page was a 404. It pointed at the relative
../CHANGELOG.md, which the Store resolves against the username, not the actor —
https://apify.com/dc83/CHANGELOG.md → 404, measured. The working public URL is
https://apify.com/dc83/clutch-b2b-intelligence/changelog (200, renders this whole file
as HTML). The README now links there.
The page listed exactly one release — v0.1.0 — after eleven had shipped. A visitor
reading it concluded the actor had never been touched since launch, which is the precise
opposite of the intended signal. Replaced with a dated table of seven releases written in
buyer language ("what changed for you"), not commit language.
Measured, for the record
…/clutch-b2b-intelligence/changelog → 200, but served with robots: noindex,follow ⇒ it
works on a human already on the page and earns nothing in Google. That is why the dated
table lives in the README (which is indexed, see [0.2.9]) rather than only behind the link.
…/clutch-b2b-intelligence/source-code → 200: the source is public too. A secret sweep of
the served bytes found no apify_api_, AIza, sk-, Bearer, owner email or IBAN patterns.
https://github.com/dc83-lab/apify-scrapers → 404 (private repo) ⇒ not linkable from the page.
[0.2.9] — the Google leg: the Store page rebuilt for search, not just for the Store (2026-08-05)
Docs-only; no runtime code changed. Gate #2 (a brand-new actor gets ~0 organic traffic regardless
of quality) has been treated so far as a problem of Apify's internal Store search, where we
rank #59/184 on clutch and are invisible on almost every target phrase. This release opens the
second, independent channel.
The measurement that made it worth doing
Fetched the live page (https://apify.com/dc83/clutch-b2b-intelligence) and read the served HTML
rather than assuming what it contains:
Fact
Reading
robots
index,follow; canonical correct
README rendering
real server-rendered HTML — every ## is a true <h2>, all the way down to the Changelog
<title>
65 rendered chars — Apify appends · Apify, pushing us past Google's ~60-char cut
<meta name="description">
truncated by Apify at 155 chars with a mid-word … ("empty res…")
JSON-LD
Organization + BreadcrumbList only — noSoftwareApplication/Product, so no price/rating rich result. Not controllable from here
our own <h3> count
zero — the 11 FAQ questions were bold paragraphs, which carry no structure
Fixed
seoTitle cut 57 → 51 chars so the full title survives Google's truncation including
Apify's · Apify suffix. seoDescription cut 157 → 147 so the meta tag stops ending in a
mid-word ellipsis. Both applied with PUT /v2/acts/{id} — these fields do not deploy with
the source.
The page claimed contactEmail was required. It has not been since 0.1.15 — it was
defaulted to a placeholder precisely because required + default are mutually exclusive and
the Store auto-tests the default input daily. Corrected in all three places it was stated.
Added — question-shaped structure, because that is what Google extracts
FAQ promoted from bold paragraphs to ### headings. Google builds featured snippets and
"People also ask" by pairing a heading with the text beneath it; bold text is not a heading.
Five new FAQ entries covering high-intent queries the page answered nowhere: Is there a
Clutch.co API? (there is none — that is the whole reason this niche exists), How many agencies
can I scrape in one run?, Can I scrape by city, country or service?, How much does it cost to
scrape 1,000?, How do I export to CSV, Excel or Google Sheets?
New section — "What can you use Clutch.co agency data for?" — five concrete use cases
(selling to agencies, shortlisting one to hire, market research, partner recruitment, CRM
enrichment). The page described the product thoroughly and the job not at all.
Three headings reworded from internal jargon into the question a buyer types: "The AI ICP-fit
artifact — what makes this different" → "How does the AI ICP-fit score rank agencies?";
"Reliability — measured, not claimed" → "How reliable is this Clutch.co scraper?" (the headline
numbers moved up into the first sentence, where a snippet can reach them); "Formats &
integrations" → "What export formats and integrations are supported?".
[0.2.8] — a per-request ceiling on block patience (2026-08-05)
Found by measurement, not review. 0.1.18 (the 0.2.7 build) was given the mandatory
post-deploy validation run and it was worse than the build it replaced — the first time in
this repo that the A/B control inverted the usual verdict. The 0.2.7 changes are all still
correct; one of them (finding E, the deeper block ladder) turned out to interact badly with the
run-level budget, and this release bounds it.
The measurement
Both builds fired on the same minute against the same proxy pool, input {}:
control 0.1.17
subject 0.1.18
wall clock
93 s
364 s
blocked requests
0 / 26
7 / 26 (blockRate 0.269)
profiles delivered
24
18
exit IPs burned
4
18
block backoff spent
21.5 s
222.7 s — run budget EXHAUSTED
The "a 403 wave arrived after the control had already finished" hypothesis was tested and
rejected: both logs put the first blocks at t+47/67 s and t+48/65 s — the same seconds. The
external 403s were identical; the only variable was how we answered them.
The whole difference is one URL. …/profile/promodo absorbed six escalating waits
(2.6 + 6.3 + 19.5 + 29.6 + 56.2 + 45.0 = 159 s) and never recovered. The run-level budget did
fire, at t+310 s — by which point every target queued behind that URL was abandoned unfetched.
Patience was measured per URL while the pot it spends is per RUN. Six block retries is the
right amount of patience for one request in isolation; it is the wrong amount when it comes out
of a 240 s pot shared with 25 other targets. One stubborn URL could legally drain almost all of
it and starve everything behind it — which is exactly what happened. The budget is a runaway
guard for the whole run, and it cannot double as a fairness rule between requests.
The ceiling ABANDONS, it never truncates. Same rule as the run budget (0.2.7): a wait that
does not fit is not taken at all, because a shortened wait followed by a real re-request from a
rotated exit IP is the sub-second burn wearing a compliance costume. Past the ceiling the URL is
reported blocked and the run moves on. Abandoning is cheap and honest — a blocked profile is
never pushed and never charged, so the buyer loses a row, not money.
A capped refusal gets a retry ceiling of 0, not maxRetries. Third member of the family that
already contains the budget latch and the abandoned Retry-After: "this URL has had all the
patience it gets" must not be implemented as a 250–500 ms generic retry aimed at the one target
refusing us hardest. At the shipped defaults attempt already exceeds maxRetries when the cap
fires, so this was invisible until maxRetries was raised — it is now pinned by a test that
fails without the guard.
60 s is sized off measured RECOVERIES, not guessed. Across Gate #1's five clean days the
worst run that recovered had slept 38.2 s. 60 s covers every recovery this actor has ever
actually made, with margin; waiting past it has never once turned into data. On the promodo shape
the ceiling stops at ~58 s instead of 159 s.
New counter blockBackoff.perRequestCapped in run OUTPUT, and BLOCK_BACKOFF_PER_REQUEST_MS in
env (floored at 1000 ms — a 0 arriving from env is far likelier a typo than a deliberate "let
one URL eat the whole run", so env cannot switch the guard off by accident; direct construction
with 0 still disables it, and that is the documented off switch).
Why this is a Store-compliance fix as well as a quality one
364 s > 300 s. Apify auto-tests every Store actor daily on its DEFAULT input and requires
SUCCEEDED with a non-empty dataset inside 5 minutes; 3 consecutive failures ⇒ an "under
maintenance" badge, +28 days ⇒ auto-deprecation. A day like 08-05 would have failed that test on
0.1.18. The earlier estimate of this risk (288 s against a 300 s limit) understated it — the
measured worst case had already crossed the line.
store-rank.mjs now records the actor's notice and isDeprecated fields every week, so the
badge is measured rather than assumed absent.
Tests
354 passing, 0 skipped (was 348). 6 new, all reproducing the measured shape: the promodo
abandon at 37.5 s of its own ladder with the run budget intact, the no-fall-through-to-generic
regression (fails without the fix), a second target getting its own fresh allowance, the
degenerate sub-one-step ceiling that must still take its first wait (capped never ships with
spentMs: 0), the 0 off switch restoring the full 97.5 s ladder, and a request that recovers
inside the ceiling paying nothing for it.
The 8 existing ladder tests disable the ceiling explicitly and say why — it is orthogonal to
the ladder, and a test that quietly absorbed it into its expected waits would stop testing the
ladder at all.
[0.2.7] — the nine deferred review findings, a reviews view, and the SEO rewrite (2026-08-05)
This release changes the fetch layer, so it RESETS the Gate #1 clean-day streak — the streak
is per-build by construction. Gate #1 itself already PASSED on 0.1.8 (5 clean days,
2026-07-31 → 08-04) and the actor is published; the streak restarting is now a monitoring fact,
not a release blocker.
Two of these findings change what a run REPORTS, both in the pessimistic direction: a refusal
whose body we could not read is now a block (it used to read as our own socket dying), and a
full-sized page that parses to nothing is now a block (it used to read as an empty profile).
Expect the measured block rate to go UP slightly on exactly the days the target is pushing back
hardest. That is the point: a metric that goes quiet under load cannot gate anything.
Fixed — external review of 0.1.8, findings A–G (src/fetch/client.js)
(A) A response body that fails to read no longer erases the response.res.text() on a
streamed reply can reject long after the status and headers arrived — a half-dead residential
proxy that sends headers and then drops the socket does exactly this. The rejection used to
escape normalizeResponse and surface as a plain transport error, throwing away the status line
we already had. On a 403 that is the entire diagnosis: the run reported a socket problem where
the target had refused us, and the Gate #1 block rate read LOW — the one direction that is not
safe, because it can pass a gate that should have failed. Status, headers and final URL are now
captured before the body is consumed. The mirror-image trap is closed too: a 2xx whose
body failed to read is treated as OUR transport error, never handed to detectBlock, whose
rule 5 would have called a 200-with-no-body an empty-shellblock and inflated the block rate
with our own proxy's stream failures.
(B) A Retry-After longer than blockBackoffMaxMs is now ABANDONED, not silently shortened.
It used to be clamped — a target asking for 900 s was re-requested after 60 s, i.e. we came back
fourteen minutes early, from a rotated exit IP, on a target that had just told us exactly when to
return. That is the polite-client rule broken by the one code path whose whole job is politeness,
and it is how a timed rate-limit becomes a durable pool-wide ban. Refusing to wait that long is
defensible; pretending we did is not. Counted separately as blockBackoff.retryAfterAbandoned in
run OUTPUT, because it is the one give-up that is not evidence of a dirty pool: it means the
target paced us, and a paid proxy tier does not fix pacing. Retry-After in its legal
HTTP-date form is now parsed as well (RFC 9110 §10.2.3); a date in the past falls back to the
ladder rather than becoming a 0 ms wait.
(C) Every retry knob must be a finite number, checked at construction.NaN was the
dangerous one and it was silent: attempt < NaN and attempt >= NaN are both false, so
maxBlockRetries: NaN neither planned a wait nor terminated the loop — an unbounded 0 ms retry
loop against a target that is refusing us, every exit IP in the pool burned in seconds, and no
diagnostic anywhere saying why. <= 0 stays legal: it is the documented "policy off" setting.
(D) A proxy handle with no proxied transport now THROWS instead of warning. The default
transport is globalThis.fetch, which drops an unknown init.proxy key silently, so that
combination means every request exits from the actor's own IP while the input schema and the
Store page both promise Apify Proxy. It used to log.warning and carry on — i.e. fail open,
and nobody reads a warning inside an unattended scheduled run: the run SUCCEEDS and a block rate
measured from the wrong IP is folded into the Gate #1 streak as evidence.
allowUnproxiedTransport:true
is the deliberate local-development opt-out.
(E) The retry ceiling is now applied to the path being retried.attempt was shared, so a
request that burned two transport retries on a dead proxy socket and then met a permanent 403 hit
the ceiling of 6 after only four block retries — 37.5 s of patience instead of 97.5 s — on
precisely the flaky-proxy runs where sockets and refusals interleave. The deep block policy we pay
for was being cut by a third exactly when it was needed, and the run then reported a block policy
it had not run. Worst case per request rises from 7 attempts to 10; the run-level block budget
(240 s, ~300 s with overshoot) plus ≤3 generic retries still bounds a single request under ~390 s
against the 1200 s schedule timeout, on runs measured at 26–52 s.
(G) blockBackoffMs is charged after the sleep happens, not before. A sleep that rejected —
an aborted run, a platform migration — used to leave the run claiming patience it never spent,
and could latch blockBudgetExhausted on sleeps that never occurred, shutting the block policy
down for every later URL in the run.
(F) deliberately unchanged. The blocked roll-up stays
block.blocked ||(!ok && sawBlock && noAnswer)
, which errs pessimistic — the Gate-#1-safe direction.
Fixed — a soft block no longer hides as an empty profile (src/main.js, src/state/store.js)
A full-sized page that parses to neither a name nor a website is now counted as BLOCKED.
Measured on the live probe (07-30 → 08-04): every "empty profile" WARN we have ever logged came
off a body of 110 / 207 / 232 / 234 KB. A genuinely dead Clutch stub is a few hundred bytes.
So a big page yielding nothing is a refusal wearing a 200 (Cloudflare's interstitial is a big
page) or a selector break — both must be visible, and neither may be filed as "the agency had no
data", the one classification that makes the block rate read low. Threshold
SOFT_BLOCK_MIN_BYTES = 50000, exported so it is testable and reviewable. New counter
profilesSoftBlocked keeps the two tellable apart; recordSoftBlock() deliberately does not
re-count the request (it was counted at fetch time), so blocked <= requests still holds and
blockedOut() is unaffected. Nothing is emitted and nothing is charged, block or not.
Added — a per-review dataset view (.actor/dataset_schema.json, src/parse/reviews.js)
Third table view, "Reviews", deferred from 0.2.6: unwind: ["reviews"] expands every review
out of its agency record into one row per review. Reviews are already bundled free inside the
profile event, so this only reshapes data the buyer has paid for.
reviewRating, an additive alias of rating on the review record.unwind lifts each
nested review's keys onto the parent row, and rating exists on both shapes — the agency's
overall score and the individual review's. Apify documents no precedence for that collision
(verified against apify/apify-docs), so renaming would have been a guess and dropping the field
would have broken the published output contract. Both values are derived from one source in
normalizeReview, so they cannot drift apart.
overview stays the first view, because first means default and output_schema.json points
the run's results at this dataset. A buyer landing on the empty AI table reads it as "the actor
returned nothing".
New always-on suite test/dataset-schema.test.js pins the three hazards this manifest carries:
no top-level fields (it would switch on validation and start rejecting pushes for a dataset
that deliberately mixes two item shapes), overview first, and every projected/labelled field
actually existing on the record it displays.
Changed — Store title, description and SEO fields rewritten for the queries we were missing
Title is now "Clutch.co Scraper — B2B Agency Directory, Leads, Reviews & Company Data", and
the description, seoTitle, seoDescription and the README opener carry the same four phrases:
agency directory, lead generation, B2B leads, company data. The old copy ranked for the
brand query and our own jargon ("ranked agency leads", "AI ICP fit") — phrases with purchase
intent that nobody types. Costs $0 and targets the queries where actors with 1–4 users win.
This is the Gate #2 (discovery) lever, not a code change.
Tests
348 passing, 0 skipped (was 330). 18 new: the abandon path and HTTP-date Retry-After, both
body-read-failure directions, the finiteness guard, the throw and its opt-out, the block ceiling
running its full ladder after generic retries, a rejecting sleep charging nothing, the soft-block
reclassification and its size gate, blocked <= requests, the reviewRating alias, and the
dataset-manifest suite.
[0.2.6] — Store page rebuilt to the measured winners' template (2026-08-04)
Documentation only — zero runtime code changed.git diff for this release touches
.actor/README.md, CHANGELOG.md and package.json; src/ is byte-identical to 0.2.5, so
Gate #1's five clean days (2026-07-31 → 08-04) still describe the code that is running. It ships
as build 0.1.10 because the Store page is served from the version's source files, and
check-runs.mjs restarts its per-build streak counter at 0 as a mechanical consequence.
The structure is not a preference. It is the section order measured across 12 live Store pages
(8 top performers + the 4 Clutch rivals) on 2026-08-04 and recorded in ops/store-page-spec.md:
six of the six top-rated Apify-official pages open on an H2 question, put price third, and
carry a how-to section, an integrations/API/MCP FAQ, and an explicit review ask. None of the
★0–1 actors in our niche does any of it.
Changed
Rewrote .actor/README.md to that template. Section order is now: what is it → what data →
how much does it cost → how do I use it → input → output → the AI artifact → reliability →
integrations → FAQ → troubleshooting → legal → changelog. Pricing moved from 7th to 3rd.
Dropped the H1 title. Apify renders the actor's title field as the page heading, so an H1
in the README duplicated it; 8/8 winners open on an H2.
Reframed the headings as the questions a buyer actually types ("How much does Clutch.co Scraper
cost?", "How do I use Clutch.co Scraper?") — the pattern every top page uses, and the one that
matches Store and Google search intent.
Added
A "How do I use it?" section — five numbered steps from finding a Clutch listing URL to
exporting the dataset. 6/6 official winners have one; we had none.
A "$5 of free credit buys 2,000 agency profiles" line, which is the honest arithmetic of our
own $0.0025 price on Apify's Free plan, plus a "why pay-per-event" note.
Four FAQ entries the template requires and we lacked: integrations, the Apify API, calling
the actor through an MCP server, and "Your feedback" — the explicit ask that is how a
zero-review actor gets review #1.
Four image slots marked in-file (IMAGE 1..4): a how-it-works diagram above the fold, the
Console input form, the output dataset table, and one AI artifact close-up. Apify has no
screenshot gallery — pictureUrl (the icon) is the only image field on an actor record and
every other visual is a markdown image in this file, so each slot needs a public URL.
Three of the four images, live (2026-08-05, build 0.1.11). Slots 1, 3 and 4 carry real
images with descriptive alt text (slot 2 followed in 0.1.13, below). Sources are HTML in
ops/assets/src/, rendered by headless Chrome at 2×
(ops/assets/render.sh), so every figure on them can be re-rendered when the measurement changes.
Hosted in the public repo dc83-lab/apify-actor-assets and served over raw.githubusercontent.com.
Apify cannot host them: an unauthenticated GET of a key-value record returns 403 for a named
store and a run's default store alike, and a token can never appear in a public README. GitHub is
also what the market does — of 13 images across 12 popular Store pages, 10 are on GitHub, and
Apify keeps apify-projects/actor-readme-images for exactly this purpose.
Dataset output schema (2026-08-05, build 0.1.12). New .actor/dataset_schema.json, wired via
"storages": { "dataset": "./dataset_schema.json" }, giving the run's Output tab two named table
views — Agencies (the profile record, website and Clutch URL as links, rating/reviews as
numbers) and AI ICP fit (the optional artifact). Deliberately views only, no fields:
declaring fields switches on dataset validation, and this dataset carries two differently
shaped item types (agency_profile, 19 keys; agency_intelligence, 9), so a strict field list
would start rejecting pushes — the one failure mode that would break fair billing by losing a
record we already decided to charge for. transformation has no filter clause, so neither view
can select by type; each therefore keeps a Record type column so a mixed run reads correctly.
A per-review view is deferred to 0.2.7 because unwind: "reviews" collides on rating and
sourceUrl and that needs a live check the publish schedule does not have room for.
Manifest aligned to what is actually live on the platform..actor/actor.json carried the
pre-SEO title/description from before 2026-08-04 while the Console carried the new ones;
three builds proved Apify does not overwrite the Console values from the manifest, but a stale
manifest is a trap waiting for the build that does. .actor/pay_per_event.json likewise still
said "Placeholder price, finalized after the reliability probe" for both events — monetization
went live 2026-08-05 12:28 UTC with prices $0.0025 / $0.035, isPrimaryEvent on the profile
event and an 80/20 split, so both event descriptions are now byte-identical to the live record.
The "Your feedback" ask now links the Issues tab instead of naming it. It is the only support
channel a buyer has, and a page that says "open an issue" without a link asks them to go hunting.
The fourth and last image, live (2026-08-05, build 0.1.13) — slot 2, the Console input form,
filled in for a real run and annotated field by field. The capture is the owner's (the browser
session is his); everything around it is ops/assets/src/02-input-form.html, in the same light
palette and type as the other three, because a raw dark-mode capture dropped between three light
graphics reads as a hole in a page rendered on white. Its three "default" claims are asserted
against .actor/input_schema.json — includeReviews true, agencyIntelligence false,
maxAgencies 100 — and $0.25 is 100 × the live $0.0025, not an estimate. The raw PNG is
gitignored here (a tracked bitmap rides along in every Apify build, and apify-deploy.mjs
uploads whatever git ls-files returns) and backed up in the public assets repo under src/,
because unlike the other three it cannot be re-rendered from HTML.
Actor output schema (2026-08-05, build 0.1.14). New .actor/output_schema.json, wired via
"output": "./output_schema.json". This is NOT the dataset schema and does not replace it: the
dataset schema describes what fields an item has, the output schema declares where a finished
run's results live and is what Console renders on the run page and returns in the run API's
output property. Ours declares two: agencies → {{links.apiDefaultDatasetUrl}}/items
(first, so the dataset stays the default view) and run report →
{{links.apiDefaultKeyValueStoreUrl}}/records/OUTPUT. Surfacing the OUTPUT record is deliberate
rather than diagnostic leakage — it carries charged next to emitted, so it is the buyer's own
receipt for the fair-billing promise, and it holds no key, token or proxy credential. Publish
checklist item, flagged red in the Console while every sibling was green.
The default input made auto-testable (2026-08-05, build 0.1.15) — caught in the publish dialog,
one click before going live. Apify auto-tests every Store actor daily on its DEFAULT input
and requires SUCCEEDED with a non-empty dataset inside 5 minutes; 3 consecutive failures earn
an "under maintenance" badge, 28 more days auto-deprecate the actor. Ours would have failed on
day one, every day, and the proof was free: POST /v2/acts/<id>/runs with {} returned 400
invalid-input: Field input.contactEmail is required. prefill does not save it — the spec is
explicit that prefill guides the UI "without affecting API functionality", and required +
default are mutually exclusive. So contactEmailleft required and gained a default
(you@example.com, also its prefill), the required array is now empty, and the field's
description asks for a real address instead of asserting one. Second failure mode in the same
test: maxAgencies defaulted to 100, and the live 30-profile schedule runs 80 s at the median
but 141 s on its worst of seven days — scaled up, a bad day lands at ~7.5 min, past the wall.
Default is now 25 (~68 s expected, ~2 min worst), which is also the better first-run
experience: a buyer clicking Start gets data in a minute for $0.0625 instead of waiting 4+ for
$0.25. test/input-schema.test.js flips accordingly — the old "contactEmail stays required" guard
is replaced by "nothing is required", "contactEmail carries a defaulted address" and a cap on
maxAgencies.default, so the next actor cannot re-introduce either failure. 323/323, skipped 0.
A THIRD failure mode, visible only by actually running it (build 0.1.16). With the schema
fixed, the {} run was accepted and still failed the test: SUCCEEDED in 3 s with 0 items,
because categoryOrSearchUrls also carried a prefill and an empty default — so the run had no
URLs and nothing to do. This is the whole argument for the empirical check over reading the
schema. The fix is deliberately in src/main.js, not another schema default: a schema default
is injected whenever a field is OMITTED, so an API caller passing only profileUrls would have
had a category injected and been charged for 24 agencies they never asked for — a fair-billing
break. runScrape instead falls back to DEFAULT_LISTING_URL only when both URL lists are
empty, i.e. only when there is otherwise no work to bill, and logs a WARNING naming the fallback.
Two fixture tests pin both halves (it fires on {}; it never fires when profileUrls is set) and
a third pins the constant equal to the schema prefill. First src/ change since 0.1.8 —
Gate #1's streak counter restarts by construction, but the gate itself already PASSED and the
change cannot touch the scrape/parse/billing path of a run that was given URLs. 326/326, skipped 0.
Dead exit IPs are now rotated out (2026-08-05, build 0.1.17) — src/fetch/transport.js. Found
by the very next {} run: it cleared the 5-minute wall (116 s) and still delivered 0 rows, on
a blockRate of 0. The listing fetched fine, then the residential exit went dead, and because
only a block burned the session all 25 profile fetches went back out through the same corpse —
75/75 retries classified transport, sessions 1 burned 0 (run Yw5MYnZBUAVsdhXHc). Fair
billing held perfectly (charged {0,0}), but the run was worthless, and on the DEFAULT input that
is a failed daily Store auto-test. Attribution was proven, not assumed (repo rule): the same
build re-run on the same {} input delivered 24 rows in 89 s, and the previous build 0.1.14 given
identical work delivered 25 in 67 s — so the transient was the road, not the code. The policy that
let the transient eat the run is the defect, and that is what changed: two consecutive THROWN
connections on one exit abandon it (deadExitThreshold, default 2 — one reset socket stays noise,
and any real response resets the count). Counted as a new deadExits stat rather than folded into
burned, because "the target refused us" and "the road died" demand opposite responses; the probe
reader prints and warns on it separately. Four fixture tests, incl. the exact live sequence.
330/330, skipped 0.
Removed
The competitor comparison table. Two reasons: 0 of the 8 top pages carry one, and ours had
gone stale — it described the deepest-schema rival as "97 users, ★3.9" when that actor is now
583 users at ★4.58. A stale comparison on a public page is a false claim, not a weak one.
[0.2.5] — Store page: real benchmark numbers, no placeholders (2026-08-04)
Documentation only — zero runtime code changed.git diff for this release touches exactly
one file, .actor/README.md. It ships as a new BUILD because the Store page is served from the
version's source files, so a README edit cannot reach buyers without one. The build number moves
0.1.8 → 0.1.9 while the actor code stays byte-identical, which means Gate #1's five clean days
(2026-07-31 → 08-04) still describe the code that is running, even though check-runs.mjs will
restart its per-build streak counter at 0.
Changed
Filled the reliability benchmark with measured numbers from the real Gate #1 window: 215
profiles across 15 unattended scheduled runs on 5 consecutive days — 98.1% success, 15/15 runs
completed, 0% of pages lost to blocks (Clutch returned a 403 on 6 of the 15 runs; the block-aware
backoff recovered every one in-run). The 4 misses are stated on the page rather than hidden,
together with the fact that they were charged $0.00.
Corrected the "30-day reliability run" claim, which was never true. The owner amended the bar
to 3–5 consecutive clean days on 2026-07-17; the page had kept the original wording. It now names
the window actually measured. Publishing an unearned "30-day" figure would have been the one kind
of defect this page cannot survive — the whole wedge is measured reliability.
Filled the capability comparison from research/clutch/competitor-teardown.md. Competitors
are deliberately NOT named: an unnamed comparison cannot decay into a false claim about a specific
peer, and undocumented capabilities are written —, never ❌.
Replaced the placeholder exampleRunInput on the actor record — it was still Apify's default
{ "helloWorld": 123 }, i.e. the first thing a buyer saw under "Input" was meaningless. It now
shows a real category run with the AI event and an ICP description, using a placeholder contact
email so the owner's real address is never published.
Set seoTitle / seoDescription / categories, all three of which were absent. Readable,
not keyword-stuffed — the teardown infers a stuffed title is what caps the deepest-schema rival at
★3.9, so copying it would copy a known mistake.
Default run memory 4096 → 1024 MB. Every probe schedule already overrode to 1024, so no
measurement described what a buyer's default run would cost; a network-bound Cheerio scraper
cannot use the extra memory. Base-event margin returns from ~48% to ~64%.
[0.2.4] — Block-specific retry backoff (deployed 2026-07-30 as build 0.1.8)
Deployed WITHOUT apify push: the Apify CLI has hung on the macOS keychain since 2026-07-29, so
this build shipped through scripts/apify-deploy.mjs (REST, token file, no keychain) — the
deploy-side counterpart of the scripts/check-runs.mjs reader. On-platform validation run
02XITXPLskZkZgxyd: 4/4 profiles, AI event 4/4 (the LLM key survived the rebuild), blockRate 0,
and blockBackoff present in OUTPUT. The block PATH itself is not exercised by a clean run —
its timing is pinned by offline vectors, and its live proof arrives with the first blocked run.
Fixed
FetchClient now backs off a blocked/rate_limited retry far slower than a transient
transport error or 5xx, instead of sharing one full-jitter policy for every retryable reason.
Every failed run on build 0.1.7 had the identical shape:
— dead on the first request (the entry listing page),
never reaching a single profile. The old policy retried a 403 at [0–500ms, 0–1s, 0–2s]
(expected total ≈1.75s, floor ≈0ms on every attempt) while burning a fresh sticky-session exit
IP on each block, so four distinct US residential IPs were spent in about two seconds before
maxRetries: 3 gave up and blockedOut correctly failed the run (observed wall-clock: 12s
against an 1800s timeout). Re-requesting milliseconds later from a rotated IP with the same TLS
fingerprint is itself a bot signature — the near-zero jitter floor WAS the bug, not the target
or a defect in parsing/selectors.
Measured 2026-07-25…30 via node scripts/check-runs.mjs: clean 07-25…27 (0/3 blocked runs/day
each day), then a step change on 07-28 (5/6 runs hit a 403, 16 proxy IPs burned), 07-29 (1/3, 4
burned), 07-30 morning (1/1, 1 run, 3 burned) — all with the same 4-IPs-in-2-seconds shape.
The block path now gets its own retry ceiling (maxBlockRetries, default 6, vs. maxRetries: 3
on the generic path, unchanged) and its own backoff: blockBackoffBaseMs (5000) doubling to
blockBackoffMaxMs (60000), half jitter (capped * (0.5 + 0.5*jitter()), floor = 50% of
nominal) instead of full jitter — intended spacing 5/10/20/40/60/60s. 7 attempts = 6 waits,
so one request's worst case is 5+10+20+40+60+60 = 195s nominal, a 97.5–195s window after
half jitter, across 7 distinct exit IPs.
Which path a response takes is decided by detectBlock, NOT by the 4xx/5xx split — worth
stating because it is the one thing about this change that surprises: detectBlock flags
403 / 429 / 503 (503 is routinely Cloudflare "under attack"), plus challenge bodies, the
cf-mitigated header and empty shells. So the generic path is thrown transport errors and
500 / 502 / 504 only, and an origin 503 now costs up to 6 patient retries rather than 3 fast
ones. Generic-path behaviour itself is unchanged (full jitter retained; a 500 still sleeps 0 with
jitter:()=>0).
Retry-After still wins when present, but on the block path it is floored at this attempt's own
ladder step as well as capped at blockBackoffMaxMs. A Retry-After: 0 or : 1 is legal and
Cloudflare does send small ones; honoured literally it would fire the next retry ~0ms later from a
freshly rotated IP — the exact burn this release exists to kill, and with 6 block retries instead
of 3 it would be worse than before the fix. The floor is the ladder step and not the flat
base, because a target that repeats a small Retry-After on every refusal (what a rate limiter
does) would otherwise pin all six waits at 5s — 30s of total spacing instead of 97.5–195s — and
then latch retriesExhausted anyway, sending the operator to a paid plan on evidence the policy
never gathered. A larger Retry-After still wins. The generic path keeps honouring
Retry-After: 0 literally (a reset socket does not care).
The ladder is indexed by BLOCK retries of the request, not by its total attempts.attempt is
shared with the generic path, so a request that burned two transport retries on a dead proxy socket
and then hit its first 403 used to enter the ladder at step 3 — a 20s first block wait instead of
the documented 5s, spending the run-level budget several times faster on exactly the flaky-proxy
runs where blocks and socket errors interleave. The retry ceiling still reads the request total,
so a mixed-cause request buys itself no extra attempts; only the spacing of its block waits moved.
A run-level blockBackoffBudgetMs (default 240000 = 4 min) bounds total time slept in block
backoff so a fully dirty pool cannot spend the run timeout sleeping through retries that will
never succeed. Once spent (latched via stats.blockBudgetExhausted), a refusal is answered once
and abandoned — no retry at all — for the rest of the run. It deliberately does NOT degrade to
the generic ceiling: the generic path is full jitter off a 500ms base and honours Retry-After: 0
literally, so a degraded block retry fires 250–500ms later from a freshly rotated exit IP. That is
the 4-IPs-in-2-seconds burn this release exists to kill, reinstated for the tail of the run — and
the budget is charged for waits on requests that RECOVER too, so a healthy-but-pressured run
latches it partway through its target list with a third of the run still to fetch (~20 more exit
IPs burned at sub-second spacing, on a run whose OUTPUT still reads as survivable). A spent budget
means the target has been refusing us for four minutes; re-sampling the pool at 250ms is the one
response known to make that worse. Consequence to read at the probe: a run carrying
exhausted: true reports a HIGHER block rate than it would have, because targets that would have
recovered on a rotated IP are now counted as blocks. Treat that run's block rate as an upper bound,
not as a clean Gate #1 measurement (ops/daily-schedule-setup.md says the same).
This is a runaway guard across MANY blocked requests, not a per-request cap and not the
escalation signal — on the backoff LADDER one request tops out at 195s, below the 240s budget.
It is not, however, unreachable on a single request, and an earlier draft of this entry wrongly
said it was: Retry-After skips both the ladder and the jitter, so a target repeating
Retry-After: 60 sleeps 60+60+60+60 = 240s and latches on block retry 4 of the FIRST request,
at requests 1, blocked 1. That shape means the target PACED us — rate limiting, not proxy-pool
reputation — so the response is to slow down, not to buy a proxy plan, and it does not count
toward the Creator-Plan trigger.
A block wait is never truncated to fit the remaining budget. The first cut of the budget did
clamp it (Math.min(waitMs, remaining)) and then retried anyway; with no floor on the truncation,
a budget one ladder step short produced a ~1ms "backoff" — the suite had pinned a 500ms one —
followed by an immediate re-request from a freshly rotated exit IP. That is the same
4-IPs-in-2-seconds burn this release exists to kill, wearing a budget-compliance costume, and it
slipped past the waits.every(ms >= floor) invariant test because that case's budget was an exact
multiple of the floor. A wait is now either taken in full or not taken at all: when the next one
does not fit, the budget latches and the refusal is abandoned. Total block sleep can therefore
overshoot the budget by at most one ladder step (only in the degenerate case of a budget smaller
than the first wait, kept so exhausted: true never ships alongside spentMs: 0). Measured
healthy runs take 26–52s, so the worst case is bounded at roughly 300s + fetch time. Note what
the budget does NOT bound: ATTEMPTS. A blocked-but-recovering run makes up to 7 network attempts
per target instead of 4, so fetchRequests, proxy bandwidth and the Gate #1 cost-per-1000 can
roughly double under pressure.
UNVERIFIED — a harvest step, not a claim. The schedules' real timeoutSecs are recorded
nowhere in this repo, and this branch was written in a ralphex run with no network, so no
statement here about "the shortest schedule timeout" would be checkable. Read them at harvest
(apify api GET v2/schedules); if the shortest is under ~600s, raise it or lower
BLOCK_BACKOFF_BUDGET_MS for that schedule. This matters more than it looks: OUTPUT is written
once, at the end of the run, so a run the platform kills for timeout publishes no blockBackoff
key at all — and losing the attribution evidence is exactly the failure this release exists to
prevent.
Be honest about what this can and cannot buy. It cannot outlast a pool-wide wave — the
2026-07-28 wave ran ~13 minutes and a bounded in-run backoff will never cover that; the run
should fail cleanly and the next scheduled run should succeed. It can win the probabilistic
case: sampling 7 exit IPs over ~2–4 minutes instead of 4 IPs inside 2 seconds, and survivors
already succeeded on attempt 4. Its real payoff is attribution: a run that still fails having
spent every block retry while the target was still refusing it
(blockBackoff.retriesExhausted: true in OUTPUT) is now unambiguous pool-wide-wave evidence —
the Creator-Plan escalation trigger — separable from "the fix never fired" (a wiring bug), which
was not distinguishable before this shipped.
A request that was refused and then died on OUR socket now reports blocked, closing a
silent-success hole in Gate #1.FetchClient.get() returned the classification of the LAST
attempt only, so a 403 / 403 / 403 / ECONNRESET request (or a terminal 500 after refusals) came
back blocked: false. RunStore then counted zero blocks for it, blockedOut() — which needs
blocked === requests — stayed false, and a run walled off by the target delivered nothing yet
exited SUCCEEDED, banking a fake clean day on the Gate #1 streak. Pre-existing, but this
release widens the window roughly fourfold: the request now stays alive for up to 7 attempts over
minutes instead of 4 in ~2 seconds, so four terminal attempts can land on a flaky residential
proxy socket instead of one. detectBlock still owns what a block is; this only decides which
attempt of a failed request speaks for it. The roll-up is narrow on two axes, both deliberate,
because it feeds the Gate #1 block rate:
the request must have FAILED — one that RECOVERS after a 403 still reports blocked: false, so
the block rate is unchanged for every run that succeeded;
the terminal attempt must have produced no answer from the target: a thrown transport error
or a 5xx. A definitive 4xx is excluded — a 403 wave that rotates onto an exit IP which then
returns a real 404 (a retired agency) is a request the target answered, and reporting it as
blocked with status: 404 both mislabels the reason and moves the profile off
profilesFailed onto the stricter 3% block bar (main.js skips recordFailed() for a blocked
fetch). 19 good profiles plus one 403 → 404 would flip a clean day to FAIL
(blockRate 1/21 = 0.048 > 0.03) and reset the 3–5 clean-day streak on a fetch nothing was
wrong with.
This RE-BASES the Gate #1 block rate against builds ≤ 0.1.7 — flag it at harvest, it is not
purely a bug fix. The same arithmetic that excludes the 404 case applies to the case now
INCLUDED: on clutch-noon-large (30 agencies → 31 recorded requests), one profile ending
403 / … / ECONNRESET used to score blockRate 0 with a fetch rate of 29/30 = 0.967 (CLEAN under
the 5% bar); it now scores blockRate 1/31 = 0.032 > 0.03 (FAIL under the 3% bar) because a
blocked fetch never reaches recordFailed(). So the change makes a clean day marginally harder to
earn during exactly the wave periods this branch exists to survive. Kept anyway, and deliberately:
the old number under-reported precisely when block pressure was HIGHEST (the more the target
refuses, the likelier the terminal attempt lands on a socket error and the request reports
blocked: false), and a metric that goes quiet under load cannot gate anything. The 3% bar was
calibrated against that under-reporting number — owner sign-off item at harvest: keep the bar at
3% or widen it, on a metric that now measures what it always claimed to.
retriesExhausted counts BLOCK retries, is compared against maxBlockRetries rather than the
live ceiling, is gated on the rolled-up blocked, and requires at least one block retry to have
happened. All four narrowings protect the same claim — the flag is the Creator-Plan escalation
trigger, i.e. a real spending decision, and every doc that reads it says "all 6 block retries
across 7 exit IPs, 97.5–195 s slept".
Counting the request TOTAL (attempt, shared with the generic path) over-reported on
interleaved runs: 2 transport retries on a dead proxy socket followed by a permanent 403 hits
the shared ceiling of 6 after only 4 block retries — 37.5 s slept — and used to latch the
trigger on a third of the evidence, on precisely the flaky-proxy runs where sockets and
refusals interleave. The retry CEILING still reads the request total, so a mixed-cause request
buys no extra attempts; it just cannot claim patience it never spent.
Comparing against the live ceiling over-reported after the budget latch: the ceiling drops to
0 for a refusal once the budget is spent, so >= ceiling would fire for a request that was
never allowed to retry at all, contradicting the exhausted flag printed on the same run.
Gating on the last attempt's classification UNDER-reported, and contradicted the block roll-up
above: a request refused through the whole ladder whose terminal attempt dies on our socket
spent every second of the claimed patience across all 7 exit IPs, and the roll-up already
counts it as a block-out. Reading the same rolled-up blocked value reports one event one way,
and carries the needed exclusions for free — a request that RECOVERS on its 7th attempt, and a
definitive 4xx the target actually answered, both report blocked: false and cannot latch it.
Requiring blockAttempt > 0 stops a misconfigured maxBlockRetries: 0 from publishing
{ spentMs: 0, retriesExhausted: true } on a bare 0 >= 0 — check-runs.mjs would print
"ceiling 0, slept 0.0s … pool-wide wave evidence" and the runbook routes that line to a paid
plan, on a policy that never ran. readTuningEnv floors the knob at 1, but FetchClient is a
public injectable collaborator; symmetric with the guard the sleep budget already carries.
It still fires when the budget runs out on the LAST of a full set of block retries, which is
genuine wave evidence.
A blockBackoffBaseMs or blockBackoffMaxMs of <= 0 now disables the block path, the same
answer a <= 0 budget already gets. _blockBackoffMs multiplies by the base and caps at the
ceiling, so either at 0 makes every ladder step 0 ms — and here the budget latch cannot even act
as a backstop, because nothing is ever spent. Left ungated, a refused request took all 6 block
retries back-to-back in milliseconds (7 exit IPs burned, worse than pre-0.2.4) and latched
retriesExhausted on a spentMs: 0 run, forging the paid-plan trigger. readTuningEnv floors
these at 100 / 1000, so this is direct-construction only — same reasoning as the sibling guards.
The escalation WARN in check-runs.mjsclaims pool-wide-wave evidence only when the run did not
deliver (non-SUCCEEDED status, or blockedOut), and now carries blocked/total inside the line.
The runbook rule has always been "retriesExhausted: true on a FAILED run", but the code asserted
the wave reading on the flag alone: with maxAgencies at 100, one target outlasting the ladder
while 99 succeed is a ~1 % block rate — under the 3 % bar, so a WARN on a run that delivered — and
the operator still read "pool-wide wave evidence, not a tuning problem" on a demonstrably working
pool. That is a false increment on a counter whose next step is spending money. The rule lives in
one exported escalationNote() shared by verdict() and line(), because a FAILED run — the one
shape where the claim IS warranted — has its WARN list discarded, so the detail block is the only
place the reading reaches a human.
Added
MAX_BLOCK_RETRIES, BLOCK_BACKOFF_BASE_MS, BLOCK_BACKOFF_MAX_MS, BLOCK_BACKOFF_BUDGET_MS
env knobs (readTuningEnv() in main.js), so retuning the policy during the probe is a
version env-var change + rebuild, not a source edit and review round. It is NOT a
set-and-go: this actor's env vars bake in at BUILD time (see 0.2.3 "Operational"), so a new
value only takes effect on the next build. Each falls back to its conservative default with a
warning on an unparseable/out-of-range value, matching every other tuning knob in this actor.
BLOCK_BACKOFF_BUDGET_MS clamps at a 1000 minimum rather than 0. There is deliberately no env
off switch.FetchClient itself treats a <= 0 budget as "block path disabled" (blocks take the
generic ceiling and full-jitter backoff, i.e. the pre-0.2.4 policy) — it must never mean "retry
deeply with no wait", which is what a 0 budget used to produce: every sleep clamped to 0ms, the
budget latch correctly refusing to forge exhausted without sleeping, and therefore all 6 block
retries fired back-to-back in milliseconds — 7 exit IPs burned instantly, worse than pre-0.2.4
and invisible in the diagnostics. Exposing that as a knob would let one env var silently restore
the burn while OUTPUT still read spentMs: 0, indistinguishable from an old build. And
MAX_BLOCK_RETRIES=1 is not an off switch either: it is weaker than pre-0.2.4 (1 retry vs the
3 a 403 used to get) and it makes retriesExhausted — the Creator-Plan escalation trigger, a real
spending decision — latch after a single refusal. Any lowered ceiling dilutes that flag, which is
why check-runs.mjs prints the ceiling and the seconds slept inside the escalation line itself.
Retune politeness with BASE/MAX; turning the policy off is a code change and a CHANGELOG entry.
blockBackoff: { spentMs, exhausted, retriesExhausted, maxBlockRetries } in run OUTPUT,
alongside the existing proxySessions ride-along — diagnostics, not billing state, so it is NOT
folded into RunStore/STATE (folding would zero the cumulative spentMs and the latched
booleans on every fold, destroying exactly the fact the next probe reading needs). Without it
the daily check cannot tell "the deep backoff fired and we still lost" (target waving, escalate)
from "it never fired" (fix not wired). The two latches answer different questions and must not
be confused: retriesExhausted = "we spent our full patience on ONE request and it was still
being refused" (per-request, reachable on the first request, the escalation trigger);
exhausted = "the run-level sleep budget ran out" (across many blocked requests, a runaway
guard, and unreachable in the requests 1, blocked 1 shape every failed run actually has).
scripts/check-runs.mjs reads it: a WARN (never FAIL — the run still survived) when
blockBackoff.spentMs > 0, naming the seconds slept; a distinct WARN when
blockBackoff.retriesExhausted === true (the Creator-Plan escalation signal) which names the
ceiling and the seconds slept in the line itself — the flag only means "wave evidence" at the
shipped ceiling of 6, so the evidence has to travel with the claim that triggers a paid plan — and a
separate, much rarer one when blockBackoff.exhausted === true; the per-day "Block pressure" trend line
now totals seconds spent in block backoff. The per-run detail line prints the block-backoff
numbers unconditionally, including on a FAILED run — a FAIL discards the WARN list, and a
failed run is precisely when the runbook says to read these. A run whose OUTPUT predates this
change (no blockBackoff key — every run on build 0.1.7 and earlier) prints no block-backoff
line at all, reading as "old build" rather than as zero, and never breaks the clean-day streak
on its own — only a FAIL does.
TDD, offline, exact-vector assertions (not ranges) in test/client.test.js: a forever-blocking
transport with maxRetries:3, maxBlockRetries:6 yields attempts === 7; a forever-throwing
transport still stops at attempts === 4 (generic path provably unchanged); with
jitter:()=>0 the recorded block sleeps are exactly [2500, 5000, 10000, 20000, 30000, 30000]
(every one of these would be 0 under the old full-jitter formula) and with jitter:()=>1 they
are the full nominal ladder [5000, 10000, 20000, 40000, 60000, 60000] summing to exactly
195000 — the two ends of the half-jitter window pinned as numbers, not prose; a 500-level error
still sleeps 0 with jitter:()=>0 (full jitter retained off the block path); a 503 and a 429
both take the block path (the split is detectBlock, not 4xx/5xx); Retry-After: 900 caps at
blockBackoffMaxMs while a Retry-After below this attempt's ladder step floors up to it, and a
target repeating Retry-After: 1 on every refusal still climbs the full
[5000, 10000, 20000, 40000, 60000, 60000] = 195000 rather than six flat 5s waits; two transport
errors followed by blocks yield exactly [0, 0, 2500, 5000, 10000, 20000] — the ladder starts at
its FIRST step and the ceiling still stops the request at 7 attempts; a 403 → 404 reports
blocked: false, blockReason: null (the target answered) while a 403 → ECONNRESET and a
403 → 500 both report blocked: true; a 0 budget disables the block path outright
(attempts === 3 on the generic ceiling, waits [0, 0], no flag latched) rather than firing 7
instant retries; retriesExhausted
latches only when the retries run out while still blocked, and provably not when a block clears
in time nor when a transport error exhausts maxRetries; the budget accumulates across separate
get() calls (it is run-level), never overspends, and cannot be forged without sleeping; once
latched the client demonstrably stops retrying refusals altogether (attempts === 1 on every
later refusal, no wait below the half-jitter block floor anywhere in the run) rather than
degrading to sub-second generic retries, and a request cut off that way does NOT latch
retriesExhausted, while one that spends its full block patience still does — including when its
terminal attempt dies on our own socket, and excluding one that recovers on its last exit IP;
maxBlockRetries: 0 latches nothing at all;
foldFetchStats leaves the block-backoff diagnostics alone while zeroing the foldable counters;
a refused request that ends on ECONNRESET or a 500 reports blocked: true with its
blockReason and its transport error both intact, while a recovered request and a 500-only
failure both report blocked: false; and a FetchClient built with no new options behaves
identically to today for transport/server_error. scripts/test/check-runs.test.mjs pins that
the escalation WARN carries ceiling 6 / slept 195.0s, and that a diluted maxBlockRetries: 2
run shows ceiling 2 / slept 7.5s instead of reading like the real trigger.
[0.2.3] — Empty-profile drop made diagnosable (deployed 2026-07-28 as build 0.1.7)
Fixed
A non-chargeable ("empty") profile is now logged with its URL and response size instead of
vanishing. Every other drop path — an off-target redirect, a parser throw — already named the
URL it discarded; this branch incremented profilesEmpty and moved on in silence. The counter
alone cannot separate the two cases that matter: a genuinely retired page (correct, and free to
the buyer) from a full page whose layout our selectors stopped matching (a regression, and
exactly the silent data loss Gate #1 exists to catch). The response size is the discriminator —
a few hundred bytes is a shell, tens of KB means the content was there and we missed it.
Found by auditing 4 days of unattended scheduled runs (2026-07-28), where 2 of 142 profiles
(1.4%) disappeared leaving nothing in the log to chase. Billing behaviour is unchanged: the
record is still dropped and still never charged. TDD, full suite 288/288 skipped 0.
Operational (no source change)
Build 0.1.6 was a REBUILD, not a code change: version env vars (LLM_API_KEY secret,
LLM_BASE_URL, LLM_MODEL) bake in at build time, so setting them on the actor did nothing
until the actor was rebuilt. .actor/actor.json carries an environmentVariables block
(@clutchLlmKey) so a later apify push re-applies the key instead of wiping it — verified
intact after the 0.1.7 push.
The icpFit branch and its stricter requireIcpFit artifact gate ran on live data for the
first time on 2026-07-28 (an icpDescription was added to the evening probe schedule): real
differentiated scores 0.8 / 0.75 / 0.4 with ICP-conditioned reasoning, grounded quotes and
confidence intact, gate passing 3/3. No code change was needed — the branch was correct, only
unexercised.
[0.2.2] — US proxy pin + blocked-out silent-loss guard + Plan B proxy-tier fallback
(deployed 2026-07-24 as build 0.1.4; hotfixed the same day to 0.1.5 — see "Fixed" below)
Added
Plan B — in-run proxy-tier fallback. The transport now takes an ORDERED list of proxy
tiers (proxyConfigurations) instead of one, starts on tier 0, and advances to the next tier
when the current one proves dirty: switchThreshold (default 2) CONSECUTIVE confirmed blocks.
This is the layer above sticky sessions — a single bad IP is still just burned and retried
inside the tier; only when a FRESH session in that tier ALSO blocks does the tier itself count
as dirty and the run fall back. A clean landing resets the streak; on the last tier the run
stays put (never silently drops to a direct, proxy-less connection). deriveProxyTiers() maps
the buyer's single input to the ordered list — RESIDENTIAL→[RESIDENTIAL, DATACENTER],
DATACENTER→[DATACENTER, RESIDENTIAL], carrying the US country pin (and every other field) onto
the fallback; a non-Apify / auto / unknown-group / multi-group input stays a single tier
(nothing safe to pivot to). buildProxyTiers() then turns that list into live handles and is
best-effort on the fallback: a tier the account cannot access is dropped (with a warning), not
fatal — so on a plan without a second accessible tier Plan B degrades cleanly to single-tier,
and it auto-activates the moment a second tier becomes accessible. (Proven on-platform
2026-07-24: the FREE plan cannot access DATACENTER — its country-pinned form throws on
Actor.createProxyConfiguration — so on FREE Plan B currently runs single-tier residential.)
Rationale (2026-07-23 root cause): Clutch block reputation lives per proxy POOL and flips by
tier over calendar time, so a fixed single tier always has bad windows. The run summary's
proxySessions now also carries switches (how many times it fell back) and tier (where it
ended) so the probe can read "primary pool went dirty, Plan B recovered" vs "every tier dirty →
escalate to a paid tier". TDD: transport 22 tests, deriveProxyTiers 8 + buildProxyTiers 6
tests, all guards proven load-bearing by temporary removal. Full suite 287/287, skipped 0.
Fixed
A run that delivered NOTHING because the target blocked us now reports FAILED, not SUCCEEDED.RunStore.summary() exposes a pure blockedOut predicate — requests > 0 AND zero rows emitted
(cumulative across both events, so a resumed run that already delivered is never failed) AND the
cause is blocks (blocked === requests or the segment already degraded). main() reads it and
calls Actor.fail(...) after persisting STATE + OUTPUT, so the full summary survives for
diagnostics. Closes the 2026-07-23 silent-loss case: a single fully-blocked entry page discovers
zero profiles and, because one blocked fetch is far below minBlockSample, used to exit
degraded:false and be counted as a probe "success". Deliberately narrow — a minority of blocks
mixed with plain 404s/empties (the buyer's dead URLs) and a partial-delivery degrade both still
pass, since neither is a total block-out.
Hotfix 0.1.4 → 0.1.5: Plan B's fallback tier crashed the entire run on accounts without a
second proxy tier.Actor.createProxyConfiguration({apifyProxyGroups:["DATACENTER"]}) THROWS
its access check on this FREE plan, which has no group by that name (
availableProxyGroups:{BUYPROXIES94952:5}
) — and the throw took down the working RESIDENTIAL primary with it.
buildProxyTiers() now makes a FALLBACK tier best-effort: an inaccessible tier is dropped with a
warning and the run continues on the primary, while the buyer's own PRIMARY tier failing stays
fatal. Re-validated on-platform: 3/3 delivered, blockRate 0, warning fired once. This defect
shipped past a green 287/287 fixture suite and was caught only by an on-platform run — the
reason every deploy in this repo now gets a live validation run.
Changed
Default proxy now pins apifyProxyCountry: "US" (prefill + default) on top of RESIDENTIAL.
2026-07-23 isolation: a bare residential config with no country served mixed-country exit IPs and
the non-US ones were 403'd site-wide — the /agencies index, the …/digital-marketing category,
AND two known-good /profile/*, all 3/3. Pinning US made the same URLs load first try (blockRate
0). The installed SDK honors the field (proxy_configuration.js: countryCode || apifyProxyCountry
→ country-US in the proxy URL), so a buyer on the default inherits a US exit instead of a 403.
A dedicated always-on guard test pins both prefill and default.
The transport now HOLDS one Apify Proxy exit IP instead of drawing a new random one per
request, and only burns it when a response comes back blocked (newUrl(sessionId) with a
session pinned across requests). The old per-request rotation was reasoned from "stateless
GETs need no cookie continuity" — correct about cookies, wrong about the pool. The shared
FREE-plan datacenter group holds ~5 addresses; once some are flagged, rotating every request
walks back onto a flagged one about as often as chance allows.
The 2026-07-20 08:00 UTC probe run measured it: blockRate 0.55, 24 retries, 24 of them
blocked, 4 of 15 agencies delivered before the degrade guard stopped the run. A matched
control — this actor's own transport, same hour, clean IP, no proxy — fetched every one of
those URLs on the first try, including the profile that was blocked 4/4 through the proxy.
The fingerprint was fine; the address was not.
Rotation is driven by the same detectBlock FetchClient uses, so a Cloudflare challenge
served under a 200 burns the session exactly like a 403 does. A thrown request (reset
socket, timeout) deliberately does not burn it — that is not evidence an IP is flagged,
and the incident's retries were 24/24 blocked, 0 transport.
Added
proxySessions: { sessions, burned } in the run summary — exit IPs used vs. lost to
blocks. { sessions: 1, burned: 0 } means one clean address carried the whole run;
sessions climbing with burned right behind it means the pool itself is dirty, which is
the signal to escalate the proxy tier rather than keep tuning code.
[0.2.0] — Data-quality batch from the Gate #1 short probe (deployed 2026-07-19 as build 0.1.2)
Four defects and one blind spot found by auditing what the probe actually produced over
2026-07-17…19 (34 unique agencies, 336 reviews) — not by reading the code. Every fix below is
verified against that real output, not only against fixtures.
Added
scrapedAt on every agency row — ISO-8601 UTC. The dataset previously carried no
timestamp at all: two runs a day apart were byte-identical, so a buyer diffing them could
not tell fresh data from stale, and neither could the probe.
reviewsIncluded — how many reviews this row actually carries, next to the existing
reviewCount (the agency's lifetime total). The two differ a lot on big agencies
(11 returned vs 362 total for SmartSites); the field, plus a direct README line, makes that
a disclosed limit instead of a silent one.
Retry attribution — fetchRetriesByReason (transport / blocked / rate_limited /
server_error) in the run summary, plus a warning log per retry naming the reason and the
URL. Retries rose 2 → 11 in 24h during the probe and the log could not say why; that is the
earliest durability signal Gate #1 has and it was unreadable. Accumulates across a resumed
run's segments like the other counters, and is restored defensively on resume.
Fixed
resolvedWebsite carried the referral tracking query on 30 of 34 agencies — the URL was
a campaign link, not the agency's canonical site, so it deduped and joined wrong. Stripped by
prefix, which is what catches Clutch's utm_source__c Salesforce flavour alongside plain
utm_source; a URL with nothing to strip is returned untouched rather than re-serialized.
location.city had the country glued on ("San Diego, United States") on 34 of 34, while
country held it correctly as well — so the field was unusable as a city key.
Review date was rendered text ("Jun 21, 2025") on all 336 reviews, while the README
promised ISO. Now normalized to YYYY-MM-DD via an explicit month table and regex — notnew Date(), which would make the result depend on the runner's timezone and shift dates by
a day. Unrecognized input passes through unchanged rather than becoming a wrong date.
Documented
companyEmail is effectively always "" — Clutch does not publish agency emails, it
routes through its own form. The field stays (the schema guarantee is that the shape never
changes) but the README now says so plainly and points to resolvedWebsite instead, rather
than letting a buyer plan an outreach workflow around a field that never fills.
Flexible input — Clutch.co category/search URLs (auto-expanded with pagination),
direct profile URLs, or both together, capped by maxAgencies.
Guaranteed-complete agency schema — every field always present; missing data becomes a
typed default (empty string / null / empty array), never a dropped key or broken record.
Resolved real website — the agency's actual site, decoded from the Clutch outbound
redirect rather than the redirect link.
Public reviews (includeReviews, on by default) — reviewer, role, project, rating,
text, date, and source URL per publicly rendered review.
Optional AI agency-intelligence artifact (agencyIntelligence, off by default) — a
"who to act on first" decision object: ICP fit, pain signals, risk flags, outreach angle,
evidence quotes with source URLs, and confidence. Works best with includeReviews on: the
artifact grounds its evidence in review text, so with reviews off most agencies yield no
artifact (free, but empty). The actor warns when run in that combination.
Buyer ICP input (icpDescription, optional) — describe the buyer you are and every
agency is scored for fit against it. Left empty, icpFit is returned as null rather
than guessed; the rest of the artifact is unaffected.
Enforced grounding — an evidence quote survives only if its text appears verbatim in a
single field we actually scraped AND its sourceUrl is a page we actually read. Paraphrases,
quotes stitched from two different fields, and invented links are dropped; if nothing
survives, no artifact is emitted and nothing is charged.
Pay-per-event billing — two events (agency_profile, agency_intelligence), each
charged strictly after a real result is stored.
Fair billing — never charged for an empty, not-found, blocked, or failed result, for
either event. A charge the platform refuses (the run's maximum charge is spent) is never
counted as a sale, and the run then stops cleanly rather than scrape what it cannot bill.
Polite fetch layer — declared contact User-Agent (from the required contactEmail),
token-bucket rate limiting, retry/backoff, and configurable Apify Proxy (datacenter or
residential).
Block detection + graceful degradation — anti-bot/challenge pages are classified and
skipped; if the block rate crosses a safe threshold the actor stops charging and degrades
cleanly rather than bill for degraded output.
Idempotency — no agency or event is emitted or charged twice within a run.
Webhooks — optional webhookUrls to POST each result as it is produced (best-effort,
non-fatal).
Operator configuration (env vars, not buyer input)
LLM (premium event):LLM_API_KEY (or OPENAI_API_KEY), LLM_BASE_URL, LLM_MODEL,
LLM_TIMEOUT_MS. No key → the AI event is skipped and nothing is charged.
Reliability-run tuning:RATE_CAPACITY, RATE_REFILL_PER_SEC (politeness vs.
throughput), BLOCK_RATE_THRESHOLD, MIN_BLOCK_SAMPLE (degradation sensitivity),
MAX_PAGES_PER_LISTING (pagination ceiling). Each falls back to its conservative default
when unset; an unparseable or out-of-range value warns and uses the default.
Notes
Prices are launch prices and may be adjusted after live reliability measurement.
The concrete stealth mechanism and proxy tier are finalized during the reliability run.
An artifact is charged only if it carries at least one verified evidence quote, and every
quote it carries is verbatim source text. The actor enforces the quotes themselves, not a
per-claim mapping from each pain signal / risk flag to a specific quote.
