# Changelog of Secretary of State Business Search - KYB, Officers, Agents (`seibs.co/business-registry-intel`) Actor

- **URL**: https://apify.com/seibs.co/business-registry-intel/changelog.md
- **Full Actor documentation**: https://apify.com/seibs.co/business-registry-intel.md

## Changelog

### 0.9 - 2026-09-27 - Reliability: shared browser, browser lane, time-aware runs, per-state coverage

- **Browser tier rebuilt around one shared Chromium per run** (`src/browser_session.py`): each state search gets an isolated context on the same browser, open pages are capped by run memory (1 below 2 GB, 2 at 2 GB, 3 at 4 GB+), each browser search is capped at 120 s (single fetches at 75 s), and a circuit breaker switches the tier off after 4 consecutive failures. Previously every browser state launched its own headful Chromium, up to four at once, which exhausts a 1 GB container on `ALL`-states runs. A private Xvfb starts when the image ships one and no display is running; `--disable-dev-shm-usage` keeps renderers alive on the 64 MB container /dev/shm; a missing patchright Chromium falls back to the Playwright build (the Dockerfile now installs both).
- **Two lanes:** browser-recipe states share the browser's page cap while HTTP registries run at full concurrency, so fast states are never queued behind slow portals. Verified locally on `states: ["ALL"]` with two companies and a 150 s timeout: every HTTP registry completed and the run finished SUCCEEDED inside its limit.
- **Time-aware runs:** the compute cap shrinks to fit the platform timeout (`ACTOR_TIMEOUT_AT`) minus a 45 s emit reserve. Searches and profile fetches still running at the deadline are cancelled; finished work is emitted and charged, unreached states become uncharged `fetch_error` records with `stage: "deadline"`.
- **Per-state coverage (new output):** `access_notes.coverage` lists each queried state with `status` (`returned` / `no_match` / `blocked` / `pending` / `not_reached`), entity count and reason, and the run's status message summarizes it. A run with no entities also pushes an `availability_notice` and costs $0.
- **Faster browser searches:** the Cloudflare-interstitial wait now checks before sleeping (no fixed 4 s idle per page), a missing results selector releases the page after 20 s instead of 60 s, and a CAPTCHA-gated search with the solver off is decided before any page opens.
- **Charging honesty:** every emitted entity is charged exactly once; the time cap no longer skips accounting for entities already built. Only the hard record cap (1,500) truncates, and entities past it are not emitted.
- Default run memory 1024 -> 2048 MB.

### 0.8.2 - 2026-06-23 - SC business-name search via CapSolver over the HTTP tier

- SC's reCAPTCHA-gated business-NAME search now works. The browser tier proved unreliable on the platform (the headful browser times out loading SC over the residential proxy), so it is retired for SC. Instead the actor solves the reCAPTCHA via the opt-in CapSolver hook (which only needs the site key + page URL, no browser) and POSTs the `g-recaptcha-response` token to `/Entity/Search` over the http tier (curl\_cffi) - the path that already works for SC. Verified live end-to-end: a single token POST returns the full results grid (Entity Name | Date | Type | Status | Incorporated State), parsed into entities with status + incorporated state, pagination rows filtered.
- New connector hooks `captcha_http_search` / `build_captcha_search` (base-class default None, SC overrides); orchestrator solves then submits over the http tier, gated on `solver_enabled()` (honest `state_pending` when the solver is off - the agent search + profiles need no solver). Requires `CAPTCHA_SOLVER_PROVIDER` + `CAPTCHA_SOLVER_KEY` env (wired via the `@CAPSOLVER_KEY` account secret).
- Fixture sc\_name\_results.html + smoke-test coverage of the name-search path.

### 0.8.1 - 2026-06-23 - Fix: entity/cluster/officer/graph records now reach the dataset

- **Critical output fix.** The dataset schema requires `scraped_at` on every record, but `build_entity_record` (entity records) and the cluster / officer\_link / ownership\_graph builders never set it. Apify rejected every one of those records at `push_data` ("Schema validation failed"), so the actor's core output silently never landed in the dataset - it emitted only the access-notes header and error records. Found by verifying SC on the deployed actor (the local smoke test's fake `push_data` skipped schema validation, hiding it). Added `scraped_at` to all four record builders + a smoke-test guard that validates real record shapes against the dataset schema so this cannot regress.

### 0.8 - 2026-06-20 - South Carolina fully-parsed (registered agent + address), 14 fully-parsed

- **SC** (South Carolina, businessfilings.sc.gov) moves from catalog `state_pending` to `coverage: full`, in response to a customer request for registered-agent + registered-agent-address lookups. Verified live (curl\_cffi, datacenter IP):
  - **Entity profile** (`GET /Entity/Profile/{guid}`) returns the **registered agent + agent address**, status, entity type, incorporated state, effective date, and the filing-history table - **no CAPTCHA**.
  - **Registered-agent search** (`POST /Entity/AgentSearch`) returns every entity an agent serves (agent name -> entities), **no CAPTCHA**. Exposed through `officer_lookup` (officer\_names) for connectors that declare `supports_officer_search`; only SC does, so no other state's behavior changes.
  - Only the business-**name** search (`POST /Entity/Search`) is reCAPTCHA-gated, so it routes through the existing browser tier + the opt-in CAPTCHA solver (AZ pattern). The agent + profile legs - the requested capability - need neither a browser nor a solver.
- Added a `state_entity_id` field to the normalized schema: the state's human-facing registration number (SC "01083688") when it differs from the portal id we keep in `file_number` (SC: the GUID URL token, so a record round-trips through `entity_profile`). Defaults null for every other state.
- The browser-recipe gate in the orchestrator now asks every connector for a recipe (each self-gates), so an otherwise-http connector can route just its CAPTCHA-gated leg through the browser tier while keeping profile + agent search on the fast http tier. Fully-parsed count: 13 + SC = **14**.
- Session warm-up (`connector.session_warmup()` + `RegistryClient.warm_session()`): SC scopes its `Entity/Profile` pages to a session that has executed a search (a cold profile GET is bounced to the search page). The actor now warms each such state's session once on the shared client before profile enrichment, so `entity_profile` by file\_number and browser-search-derived rows alike fetch the registered agent + address. Verified live end-to-end for both `officer_lookup` (agent name -> entities + agent address) and `entity_profile` (GUID -> agent + address). Fail-soft and default-None for every other state.
- Fixtures: sc\_profile.html, sc\_agent\_results.html (live captures). Smoke test +4 SC cases.

### 0.7 - 2026-06-12 - Cloudflare defeated: PA + MI fully working (13 fully-parsed)

- The browser tier now opens a **stealth-patched browser (`patchright`, bundled) in headful mode**, which **passes the Cloudflare/Imperva managed challenges** that block plain headless Chromium (verified live). For Cloudflare-fronted JSON APIs it calls the API **from inside the warmed page** via `fetch()` so the request carries `cf_clearance` + the real browser TLS (a Playwright APIRequestContext does not - it 403s).
- **PA** (Pennsylvania, ~3M entities) and **MI** (Michigan) are now `coverage: full`, validated end-to-end through the actor (PA: bizfile POST API in-page; MI: webSearch GET API in-page; both return real entities, MI with agents). Fully-parsed count: 11 http + PA + MI = **13**.
- Added `BROWSER_HEADLESS` override (default headful, required to pass Cloudflare); GET support in the api-recipe mode; do not override the browser user\_agent (keeps patchright's stealth fingerprint consistent). `via: "browser"` tags browser-sourced rows.
- The remaining 12 Cloudflare/SPA states (AK, AR, IL, MA, MD, MN, NC, NM, NV, OH, OK, WA) now **pass the challenge** too - they only need their per-state search API/selectors captured + a ~20-line connector (copy PA/MI). The 9 CAPTCHA/login states (AZ, GA, NE, SC, WY, DC, DE, HI, VA) still need the opt-in solver / a login / a bulk file and stay fail-soft.

### 0.6 - 2026-06-12 - Browser tier v2: CDP connect + recipes + opt-in CAPTCHA solver

- Browser tier now connects to an operator-supplied warm anti-detect browser over CDP (`browser_cdp_url` input or `BROWSER_CDP_URL` env) - inheriting its session/fingerprint to pass the Cloudflare/Imperva managed-challenge states (PA, OH, NC, MI, NV, ...) that block a plain headless Chromium. Falls back to launching headless when unset.
- Recipe-driven `browser_search`: navigate -> fill -> (optional CAPTCHA solve) -> submit -> capture the search XHR JSON or rendered HTML, plus an `api`-after-warmup mode (PA replays the bizfile JSON API on the cf\_clearance'd session). PA + AZ have specific recipes; the other passable SPA states use a generic search-and-extract recipe (rows tagged `parse_confidence: "generic"`).
- Opt-in CAPTCHA solver hook (`captcha_solver.py`, off by default): set `CAPTCHA_SOLVER_PROVIDER` (2captcha|capsolver) + `CAPTCHA_SOLVER_KEY` env secrets to clear the per-search-CAPTCHA states (AZ worked example). Genuinely gated states (CAPTCHA/login/Enterprise-reCAPTCHA: GA, NE, SC, LA, WY, DC, DE, VA) stay fail-soft with documented reasons and make no doomed generic attempt.
- `access_notes.browser_tier` reports whether the CDP endpoint + solver are configured. Validated end-to-end on the real runtime: OR returns data; PA/OH execute the browser tier (0 rows headless without the warm browser -> fail-soft); AZ reports "CAPTCHA-gated, solver disabled"; GA stays pending. The actor now ships `playwright` so the CDP path works in production.

### 0.5 - 2026-06-12 - Batch A integration (11 fully-parsed) + form-bootstrap + browser-tier validation

- Added NJ as a fully-parsed registry (validated live, real fixture): New Jersey DORES free Business Name Search via a new GET-token -> POST **form-bootstrap** flow (`RegistryClient.fetch_form_flow` runs the GET + POST on one cookie-persistent session so the `__RequestVerificationToken` round-trips). NJ free path returns name/id/city/type/formation date; status/agent/officers are paid and kept out of scope (like TX/DE). Fully-parsed set is now CA, NY, FL, TX, CO, CT, OR, WI, NJ, ID, ND (11).
- Integrated Batch A's catalog corrections (validated live against the portals): AZ (eCorp retired -> Arizona Business Center SPA, **search CAPTCHA**), MI (COFS SearchApi decommissioned -> MiBusiness Registry, now **Cloudflare**-gated), IL (JS/JSP SPA, not ASP.NET), WA (legacy JSON API dead -> CCFS SPA + token), MA (search POST anti-bot-challenged), NC (now Cloudflare), VA (reCAPTCHA v3 Enterprise -> use the weekly SCC bulk file). MI moved from `http_json` to `browser_required`.
- **Browser tier validated end-to-end**: the actor's Playwright `browser_fetch` fetched FL Sunbiz headless and parsed 20 real entities. But no `browser_required` state is cleanly automatable headless (PA/OH/GA/NC/MI Cloudflare managed challenge blocks headless Chromium; AZ search CAPTCHA; VA/NV/NE/SC/LA/WY reCAPTCHA/Imperva; DC login) - per the no-CAPTCHA/no-login posture these stay fail-soft with documented reasons.
- Status-rule additions (Batch A): `hold` -> suspended (OH), `dead` -> dissolved (OH), `past due` -> suspended (IN).

### 0.4 - 2026-06-11 - 10 fully-parsed states (parallel spec integration)

- Added 2 fully-parsed registries from the parallel research-spec docs, each validated against a real captured fixture: OR (Oregon Business Registry, Socrata `tckn-sxa6` - multi-row-per-entity, grouped by registry\_number to assemble agent + principal + mailing + authorized representatives) and WI (Wisconsin DFI Corporate Records - plain GET HTML, no ViewState). Fully-parsed set is now CA, NY, FL, TX, CO, CT, OR, WI, ID, ND (10).
- Integrated spec verdicts for the remaining states with honest, specific blockers: reclassified AR/MD/MN/NE/OK/WY to `browser_required` (SPA / Imperva / CAPTCHA / login), corrected portal URLs (KS -> sos.ks.gov/eforms, UT -> secure.utah.gov/bes), and recorded why each `http_json`-pending state can't be name-searched (IA Socrata 404 post-migration; MI TLS; MS Kendo filter returns the full DB; MT API 500s).
- Captured fixtures: or\_socrata\_search.json, wi\_html\_results.html (+ id/nd/ct from 0.3).

### 0.3 - 2026-06-11 - More fully-parsed states (8) + honest coverage tiers

- Added 3 fully-parsed registries, each validated against a real captured fixture: CT (Connecticut Business Registry, Socrata open data), ID (Idaho SOSBiz) and ND (North Dakota FirstStop) - both on the same "bizfile" vendor JSON API as California, via a new parameterized `BizFilePlatformConnector`. Fully-parsed set is now CA, NY, FL, TX, CO, CT, ID, ND (8).
- Reclassified catalog states by *why* their parser is pending after live probing: 14 `browser_required` (Cloudflare JS-challenge or SPA - need the Playwright residential tier: AK, AZ, DC, DE, GA, HI, NC, NM, NV, OH, PA, SC, VA, WA), 27 `http_html` ViewState-pending (need a per-state POST+ViewState builder), 2 `http_json` pending (MI TLS, MT API 500s). Per-state `cost_note` now states the specific blocker.
- Added real-fixture smoke tests for the bizfile platform (ID, ND) and CT Socrata; fixtures live in `tests/fixtures/`.

### 0.2 - 2026-06-11 - National coverage + anti-bot escalation

- Three-tier anti-bot escalation ladder in `registry_client`: httpx (datacenter) -> curl\_cffi Chrome TLS impersonation (residential) -> optional Playwright headless browser (residential) -> fail-soft `fetch_error`. Defeats the TLS/JA3-fingerprint WAFs registries use (verified: FL Sunbiz 403s plain httpx but returns 200 + real data via curl\_cffi from the same IP).
- Dual proxy provisioning (DATACENTER for the httpx pass, RESIDENTIAL for escalation), auto-selected per challenge. New `use_browser_fallback` input (default on). Per-tier escalation telemetry in `access_notes.anti_bot_escalation`.
- National coverage: all 50 states + DC catalogued. Added Colorado as a fully-parsed Socrata connector. Fully-parsed set is now CA, NY, FL, TX, CO; the other 45 jurisdictions are catalog-registered with correct access method/anti-bot/proxy and the escalation pipeline wired (parser pending -> documented `state_pending` note). New `states: ["ALL"]` / `all_states` to query every jurisdiction.
- Fixed the FL Sunbiz search + detail parsers against live HTML (class-attributed table cells, corporationName block, registered-agent and officer/director sections).
- Confirmed city-level queries resolve via the state connector (Miami -> FL/Sunbiz); no separate city registries exist.

### 0.1 - 2026-06-11 - Initial release

- Four modes: `entity_search`, `entity_profile`, `officer_lookup`, `ownership_graph`.
- Five state connectors: CA (bizfileOnline JSON), NY (DOS Public Inquiry JSON + token bootstrap), FL (Sunbiz HTML), TX (Comptroller HTML), DE (opt-in, browser-required, fail-soft).
- Normalized cross-jurisdiction entity schema (name, file number, status, type, formation date, registered agent, officers, addresses, filing history, source URL) with canonical status/type vocabularies.
- KYB value layer: cross-state entity resolution + `entity_cluster` rollups, officer-to-company linking with `multi_entity` flag, and an inferred ownership/association graph (shared officer/agent/address/name-root edges + connected components).
- Fuzzy matching via rapidfuzz when present, stdlib `difflib` fallback (no hard dependency).
- Graceful access handling: anti-bot/CAPTCHA detection -> fail-soft `fetch_error` with per-state access notes; `access_notes` record with the live access matrix on every run; Delaware gated off by default (never spends its per-search fee silently); proxy auto-selection (RESIDENTIAL when DE active, else DATACENTER).
- Cost-control: pre-flight caps, `_RunBudget` guard, demo-mode soft-fail, hardcoded PPE prices.
- Monitor mode (scheduled-run delta digest of new filings / status changes / dissolutions + Slack webhook), next-actions, and OUTPUT artifacts.
- Logged-out, public-data-only posture on government public records; PII minimized (name + title + business address only); association graph labeled a lead, not a beneficial-ownership determination.