# Changelog of Clutch.co Agency Leads Scraper (`devilscrapes/clutch-agency-leads-scraper`) Actor

- **URL**: https://apify.com/devilscrapes/clutch-agency-leads-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/devilscrapes/clutch-agency-leads-scraper.md

## Clutch.co Agency Leads Scraper — Changelog

### 0.1.2 — 2026-10-02 (NOT YET PUSHED — see note below)

- Triage trigger: daily fleet report flagged a 30-day customer success rate
  of 55% (13 of 29 runs failed). This Actor was NOT QA-flagged and showed
  zero of our own `recent_failures`, so the failures are customer runs we
  cannot read the logs of (see
  `reference-customer-run-logs-unreadable.md`) — reproduced locally
  against the live prefill (`category=digital-marketing`) instead.
- **Reproduction result: could not reproduce a failure against current
  live Clutch markup.** Fetched `/agencies/digital-marketing` (page 1 and
  2\), `/agencies/digital-marketing/new-york`, and two invalid
  category/location slugs directly from this machine: 4x HTTP 200 (95 rows
  parsed on the base listing, 0 skipped, 0 `ResultRow` validation errors),
  2x clean HTTP 404 on invalid slugs. All markup tokens the parser depends
  on (`provider__verified-mark--premier`/`--basic`, `ppc_item--element`,
  `sponsor`, `data-clutch-pid`) are present and unchanged. `apify run`
  locally against the real prefill (`maxResults=5`, no proxy) completed
  SUCCEEDED, 5/5 real leads written with fully populated fields. This
  rules out site/markup drift, pagination end-condition bugs, and 404
  handling as the cause.
- **Hardening applied anyway**: `src/scraper.py` `_emit_rows()` previously
  called `build_result_row()` with no fault isolation — any exception
  while building a single row (a future markup-drift-triggered
  `ResultRow` `ValidationError`, for example) propagated out of `run()`
  and crashed the *entire* page/run, discarding every other
  already-parsed row on that page. This is the fleet's documented #1
  low-success-rate pattern (`reference-fleet-fault-isolation-pattern.md`)
  and the explicit top suspect for this triage, so it's fixed
  defensively even though no failing row was observed live today: one bad
  row now logs a `WARNING` (with provider ID + exception) and is skipped;
  every other row on the page/run still ships. Added
  `test_run_isolates_a_single_row_build_failure` (red before the fix,
  green after) plus a shared-test-suite run confirming zero regressions
  (71/71 local tests green, ruff clean, pyright clean on `src/`).
- **Most likely real root cause (unresolved, needs human + cloud
  access)**: per `docs/specs/clutch-agency-leads-scraper/notes.md`
  (2026-08-10 entries), the prior fixer cycle already root-caused and
  shipped build 0.1.1 for a very similar symptom (Clutch's WAF 403-ing a
  meaningful fraction of attempts, including — in that cycle's cloud QA —
  the direct-egress fallback that originally cleared 9/9). That fix is
  the version currently live and in-sync. The 55% rate in today's report
  is consistent with that prior finding never having been fully resolved
  (QA closed it out as "qa-blocked," not "fixed"), not with a new
  regression. **This fixer cycle could not re-verify or extend that
  investigation**: the org's Apify billing is 403'ing ("too many
  outstanding invoices," a stale flag per
  `reference-apify-invoice-block-read-inbox.md`) and usage is $53.65
  against the FREE plan's $5 cap (resets 2026-10-18) — `apify push`,
  `apify call`, and every cloud run are hard-blocked this cycle. **Not
  pushed.** The fault-isolation hardening above is local-green only;
  cloud verification is explicitly deferred to the next cycle with
  platform access. `.actor/actor.json` `version` intentionally left at
  `"0.1"` (unchanged) since no new build is being tagged this cycle.

### 0.1.0 — 2026-08-10

- **Fix**: customer 30-day success rate had fallen to 55% (5/11 FAILED),
  4 of the last 4 public runs failed. Root-caused via 11 real cloud repro
  runs (build 0.0.2, `DevilScrapes~clutch-agency-leads-scraper`): direct
  (unproxied) fetches cleared Clutch's WAF in 9/9 attempts, but fetches
  routed through the FREE-tier `BUYPROXIES94952` proxy group (only 5 IPs)
  exhausted all 5 retries and failed loud in 3/5 attempts — the shared
  pool gets 403'd faster than the platform's own direct egress IP.
- `src/scraper.py`: `_fetch_page` now spends at most `PROXY_ATTEMPT_BUDGET`
  (3) of its 5 retry attempts on a configured proxy, then falls back to a
  direct fetch for the rest of the budget instead of exhausting all
  retries on a stale/blocked IP pool. Still fails loud (non-zero exit,
  `ERROR` log) if every attempt — proxied and direct — is blocked.
- Added `tests/test_scraper.py` coverage for the proxy-budget fallback,
  the proxy-not-configured no-op path, and the still-fails-loud case
  when every attempt (including the direct fallback) is blocked.
- **Not yet verified as fully resolved.** Pushed as build 0.1.1; cloud QA
  (6 samples across 2 categories) still saw ~1-in-2 runs exhaust every
  attempt, including the new direct fallback — see
  `docs/specs/clutch-agency-leads-scraper/notes.md` 2026-08-10 entries for
  full evidence and the open confound (session/volume-based WAF escalation
  from this fixer's own repro traffic vs. a genuinely tighter block).
  Needs a clean re-test before this is considered shipped-and-verified.

### 0.0.1 — 2026-07-30

- Scaffolded: `.actor/` (actor.json, input/output/dataset schemas, PPE,
  Dockerfile), `src/models.py` fully implemented (`ActorInput` + `ResultRow` +
  `ServiceMixEntry` per spec, kebab-case slug validation, bounds), placeholder
  `src/main.py` (boots, validates input, pushes one `{"placeholder": true,
  ...}` row, exits — no real crawler yet), `src/scraper.py` stub (not wired
  into `main.py`), tests, README skeleton.
- Real implementation (`src/parser.py` + `src/scraper.py`'s `run()`) is the
  next step per `docs/specs/clutch-agency-leads-scraper/design.md` (T01-T08)
  — wire format is already live-confirmed, no recon blocker.