# Changelog of DirectIndustry Industrial Suppliers Scraper (`devilscrapes/directindustry-industrial-suppliers-scraper`) Actor

- **URL**: https://apify.com/devilscrapes/directindustry-industrial-suppliers-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/devilscrapes/directindustry-industrial-suppliers-scraper.md

## DirectIndustry Industrial Suppliers Scraper — Changelog

### 0.1.1 — 2026-09-21 — Unshelve verification pass: fix invalid memory range, confirm the budget/concurrency fix holds

- **Verified, did not re-derive**: 0.1.0's run-budget + bounded-concurrency
  fix (below) was already on `main`, already pushed as build `0.1.1` on
  Apify, and already regression-tested (`tests/test_scraper.py::
  test_run_stops_before_second_seed_when_budget_already_exceeded`,
  `::test_run_stops_mid_walk_once_budget_trips_and_keeps_rows_already_
  yielded`, `tests/test_main.py::test_main_reports_truncated_status_when_
  budget_exceeded_but_rows_landed`). Re-ran the full suite (76 passed),
  `ruff` and `pyright` clean; nothing there needed changing.

- **Found while re-checking `.actor/actor.json`**: `defaultMemoryMbytes`
  was `4096` but `maxMemoryMbytes` was `2048` — a default *above* the
  configurable ceiling, left behind by the 2026-09-19 "every QA run
  measures a memory config no customer runs" finding, which raised the
  default to match the platform's real 4096 MB but never touched the
  range around it. Raised `maxMemoryMbytes` to `4096` so the manifest is
  internally consistent again; would likely have broken (or silently
  clamped) the next `apify push`.

- **Cloud evidence checked, not re-run**: the last cloud run on this
  build (`0Gx63vg1u2JEq8vBz`, 2026-09-19T09:27) FAILED — but on a seed-level
  403 exhausting all 5 retries (`SeedBlockedError`, REQ-7's fail-loud path
  working as designed), not a timeout and not a code defect. No cloud run
  has yet measured the fixed code's actual wall-clock/cost at the shipped
  prefill; that is the orchestrator's cloud-QA step to run next.

- **Root cause (measured on Apify the same day)**: every fetch — seed,
  product-type listing page, AND every individual product-detail page —
  was paced `PAGE_DELAY_S=15s` apart **serially, one at a time**, with no
  concurrency. At full prefill depth (3 seeds x up to 50 items each =
  up to ~159 total fetches), that's ~40 minutes of pure pacing delay
  alone, before any real HTTP latency. Four cloud runs the same day:
  0 rows/TIMED-OUT, 9 rows/SUCCEEDED, 75 rows/TIMED-OUT at exactly 30
  min, 100 rows still RUNNING 37+ min in — not crashing, just far
  slower than any sane run budget.

- **Fix 1 — bounded concurrency**: `DETAIL_CONCURRENCY=3` on
  product-detail fetches only (the independent, per-item class of
  request) via a small pool of independent `DirectIndustrySession`
  instances. Category/product-type navigation fetches stay
  sequential/paced — the fewer, WAF-sensitive discovery steps the live
  probe actually proved safe at the 15s cadence; each concurrent detail
  batch still shares one `PAGE_DELAY_S` pause, so the per-15s-window
  request rate triples rather than going unbounded.

- **Fix 2 — run-wide wall-clock budget** (`src/run_budget.py`, same
  shape as `cars-com-listings-scraper`'s 0.8.0 / `leboncoin-france-
  cars`'s 0.7.1 fix): derives the crawl's deadline from the REAL
  platform kill clock (`Actor.configuration.timeout_at`), not a guessed
  constant. Threaded through every crawl-loop boundary; once the
  deadline passes, the walk stops at the next checkpoint and the run
  ends **SUCCEEDED** with whatever rows it already has — never
  TIMED-OUT. `RunStats.budget_exceeded` drives an honest partial-result
  status message instead of a silent "Done".

- Expected wall-clock at full prefill depth (3 seeds, 50 items/seed, 5
  pages/product-type): ~15 minutes, comfortably inside both the 20-minute
  local fallback budget and the platform's real deadline minus its
  300s safety margin.

### 0.0.1 — 2026-09-19

- Initial scaffolded release. `src/main.py` is a placeholder that reads
  input, logs it, and pushes one `{"placeholder": true}` dataset row —
  no real scraping yet. See
  `docs/specs/directindustry-industrial-suppliers-scraper/tasks.md`
  (T02-T09) for the implementation plan.