importyeti-bol-scraper

Apify Actor for ImportYeti US customs bill-of-lading data: recent sea shipments (who ships what to whom) for any US importer, straight from the /company/{slug} page over plain HTTP with a curl_cffi chrome131 TLS profile — no browser, no proxy, no key (see docs/FINDINGS.md for the lane probe, including the paid-key verdict on the internal data.importyeti.com/v1.0 API).

Input

{"companies": ["apple"]}
{"companies": ["apple", "tesla"], "maxResults": 30, "includePartners": true}
{"companies": ["https://www.importyeti.com/company/walmart"]}

• companies (default ["apple"]): 1–50 ImportYeti slugs (the /company/<slug> path); bare URLs tolerated, deduped, case-insensitive. Omitted/empty input scrapes the default so the Store's daily empty-input auto-test survives (issue #188).
• maxResults (1–200, default 50): cap on BOL records per company — the page's "Most Recent Sea Shipments" table carries at most 50.
• includePartners (default false): detail mode — also emits one summary record per row of the company's Suppliers table.
• requestDelaySecs (0.5–10, default 1): pause between companies.

Output

One dataset item per bill-of-lading row (recordType: "bol"): recordKey (www.importyeti.com:bol:{slug}:{bolNumber}), companyName/companySlug, role (importer — /company/ pages are US importers of record), date (ISO), bolNumber (house) + bolMasterNumber, isFreeBol, carrier (from the tracking-link logo: ZIM, Maersk, HMM…), supplierName/supplierSlug/supplierCity /supplierCountry, weightKg, quantity + quantityUnit, containerCount, lane (Eu -> US Transatlantic), estimatedFreightCost + freightCurrency (USD), company context (companyTotalShipments, companyTopOriginCountry, companyAddress), sourceUrl, sourceQuery, source, scrapedAt.

Detail mode adds recordType: "partner" records keyed www.importyeti.com:partner:{slug}:{supplier-slug} with totalShipments, activityStart/activityEnd (YYYY-MM window) and productDescriptions (ImportYeti's category legend), BOL-only fields null. Supplier names the source prints as "Missing in source document" normalize to null (country/flag kept). Company-level data only — no individual PII (the page's contact "unlock" table is not scraped).

How it works

1. GET https://www.importyeti.com/company/{slug} through curl_cffi AsyncSession(impersonate="chrome131") — the chrome131 TLS fingerprint passes the site's Cloudflare check where plain HTTP clients get 403 (verified 2026-09-04; lane origin: a competitor README, see FINDINGS).
2. On 403/429/503 or a challenge page: one retry through the Apify RESIDENTIAL proxy (fallback lane, billed) before failing.
3. Next.js SSR HTML is parsed with regexes (tables are stable): JSON-LD for company identity, the "Most Recent Sea Shipments" table for BOL rows, the Suppliers table + per-country tables for detail mode and totals.

Run tests

$PYTHONPATH=src python3 -m unittest discover -s tests -v

Fixtures under tests/fixtures/ are live captures from 2026-09-04 (apple, tesla). ruff check + ruff format clean.

Cost

One ~750 KB HTML fetch per company, no proxy on the default lane: 2 companies + detail mode = 74 records in ~3 s — ≈ $0.0003/run datacenter egress. Cloudflare rotating the chrome131 profile is the documented kill criterion (issue #35); the residential fallback keeps the Actor alive at higher COGS if that happens.