# Changelog of Manta Business Directory Scraper (`devilscrapes/manta-business-directory-scraper`) Actor - **URL**: https://apify.com/devilscrapes/manta-business-directory-scraper/changelog.md - **Full Actor documentation**: https://apify.com/devilscrapes/manta-business-directory-scraper.md ## Manta Business Directory Scraper — Changelog ### 0.1.2 — 2026-08-31 - Tuned: `TRANSPORT_FAULT_MAX_ATTEMPTS` 3 -> 6, measured against 5 cloud verification runs of build 0.1.3 (Apify build tag; the 0.1.1 fix above). 3/5 SUCCEEDED, but 2/5 FAILED purely on `NS_ERROR_PROXY_CONNECTION_REFUSED` streaks — one run hit 3 transport faults in a row with ZERO bot-wall responses, the other hit 3 transport faults plus 1 genuine 403. Across all 5 runs: 8 of 12 total attempts were transport faults (a ~67% per-attempt rate against the RESIDENTIAL pool that morning), and the one large run (maxResults=120) needed 3 bot-wall retries AND 2 transport retries (5 relaunches) before succeeding — direct proof the 0.1.1 split-budget fix recovers runs the OLD shared budget of 4 would have failed outright. A budget of 3 was still measured too tight for the transport class alone; 6 gives ~91% odds of clearing that observed fault rate (`1 - 0.67**6 ≈ 0.91`) while remaining a bounded retry count (10 browser relaunches worst case, up from 7). - Both failed runs' final `set_status_message`/log correctly read `"proxy/transport faults exhausted retries — never reached manta.com"`, not `"bot-wall exhausted"` — confirms the 0.1.1 message-honesty fix holds under real field conditions, not just unit tests. ### 0.1.1 — 2026-08-31 - Fixed: the retry driver in `src/main.py` (`_run_with_retries`) shared ONE `RETRY_ATTEMPTS=4` budget between two unrelated failure classes — a genuine bot-wall response (`S.BotWallError`, manta.com answered with 429/403/a challenge page) and a proxy/transport fault (`B.RECOVERABLE_BROWSER_ERRORS`, most commonly Camoufox's `Page.goto: NS_ERROR_PROXY_CONNECTION_REFUSED` — the Apify residential proxy refusing the CONNECT tunnel, before manta.com is ever reached). Two customer-facing runs on build 0.1.2 (`hIZHkES1nOsjFv4xt`, `Y0sriBWw6IzKtRKuY`, 2026-08-31 morning) each hit the pattern 429 / PWError / 403 / PWError — 2 genuine bot-wall hits and 2 pure network faults — exhausted the shared budget at attempt 4, and both were reported as `"bot-wall exhausted retries"` even though manta.com only ever answered twice. Measured 30-day customer success rate was 60% (3/5) on this pattern. `_run_with_retries` now tracks `bot_wall_attempts` and `transport_attempts` against independent budgets (`BOT_WALL_MAX_ATTEMPTS=4`, `TRANSPORT_FAULT_MAX_ATTEMPTS=3`) — a flaky proxy exit can no longer spend the genuine bot-wall retry allowance or vice versa, and the final `set_status_message`/log line reports whichever class actually exhausted its own budget (`"proxy/transport faults exhausted retries — never reached manta.com"` vs `"bot-wall exhausted retries (REQ-4)"`) instead of always naming bot-wall on any exhaustion. A run is still failed loud in either case — this does not widen the success condition, it corrects the failure message and stops one fault class from starving the other's retries. Regression tests added (`test_main_transport_faults_do_not_consume_the_bot_wall_budget`, `test_main_transport_exhaustion_reports_transport_not_bot_wall`). - Investigated the `LeakWarning: heavily recommended that you pass geoip=True` Camoufox logs on every launch — confirmed NOT implicated in either failure mode observed (both were proxy-CONNECT-layer or app-layer, before fingerprint/geolocation coherence matters) and left off, per the documented idealista/vrbo `InvalidIP`/`InvalidProxy`/`LocaleError` crash risk of enabling it alongside a proxy. ### 0.1.0 — 2026-08-31 - **Engine swap: curl-cffi -> Camoufox.** This Actor was SHELVED with the verdict "a clean 200 is a JS-only page shell — no proxy tier can make an HTTP-only scraper work here." That verdict was true and narrow: it never named a browser engine. A throwaway `src/recon_camoufox.py` cloud probe (runs `2BAS0DTelq7bVgBDA`, `Pi1ZSgDtUGQPQ0V52`) proved Camoufox clears manta.com's Cloudflare Managed Challenge and gets real, server-rendered business rows — the "JS-only shell" was a page manta.com only fully renders for a browser that runs its JS, not a permanent wall. - Fixed: the search URL shape `client.py` always guessed (`?search=&search_source=business_directory®ion=`) was simply wrong — it reached manta.com fine (Cloudflare cleared, HTTP 200) but rendered the site's own "We encountered an error while performing your search." state. The real shape (read off the site's own outbound navigation after Camoufox filled and submitted the homepage search form) uses full state/country **names** (`state=Texas`, not `TX`), `search_source=nav`, and a `pg` param that's 1-indexed and omitted on page 1. See `src/config.py`'s module docstring and `docs/specs/manta-business-directory-scraper/notes.md` for the full trail. - Replaced `src/client.py` (curl-cffi `MantaSession`) with `src/browser.py` (Camoufox launch/proxy/fetch glue, same warm-up-then-goto pattern proven on `idealista-property-scraper`/`vrbo-vacation-rentals-scraper`) and `src/scraper.py` (pagination orchestration + retry boundary, replacing the old retry loop inside `main.py`). - Rewrote `src/parser.py`'s selectors against the real server-rendered markup (Tailwind-utility-class HTML, no BEM-style component names) — the previous selectors (`div.listing-card`, `h2.listing-card__name`, etc.) were pending-live-recon placeholders that never got to run against real markup before this Actor was shelved. List-view rows confirmed to never expose SIC code / employee / revenue estimates / founding year (paid-detail-page-only fields) — those stay `null`, not conditionally parsed. - `page_size` now defaults to 25 (the site's own maximum, per its results-per-page `