# Changelog of TikTok Ad Library Scraper (`devilscrapes/tiktok-ad-library-scraper`) Actor - **URL**: https://apify.com/devilscrapes/tiktok-ad-library-scraper/changelog.md - **Full Actor documentation**: https://apify.com/devilscrapes/tiktok-ad-library-scraper.md ## TikTok Ad Library Scraper — Changelog ### 0.4 (build 0.4.1, expected) — 2026-09-19 — T26: remove the dead `fetchAdDetails` switch, pin proxy country, rehab for publish **Rehab task** (this Actor was in `scripts/publish_actors.py`'s `SHELVED` dict): "`fetchAdDetails` is a dead switch on 0.3.1 — `/details` 421s on every ad even after a fresh re-mint, shipping 3 advertised fields null on 5/5 rows." Everything else about the Actor was healthy — build 0.3.1 had 4 SUCCEEDED / 1 FAILED, PPE pricing set, and a FAILED run bills the customer $0.00 (verified 2026-09-18: exit 91, `chargedEventCounts {'actor-start': 0, 'ad-result': 0}`). **Outcome A tried first, and it made things worse, not better.** Two header sets were now tried against `/details` across this Actor's history and both failed differently: T21/T22 (2026-09-12) found a freshly re-minted `/details` token still 421s on replay via curl-cffi. This session (T26) tried adding the browser's `cookie` header to the replay set too — a plausible, previously-untested gap in `REPLAY_HEADER_NAMES` — and pushed a real cloud build (0.3.2) to test it with `fetchAdDetails: true`. Result, run `bJWdvpbbqflogeRDV` (10/10 ads): the lazy per-page mint now failed to observe a `/details` request AT ALL on 9 of 10 ads ("No /details request observed while minting"), a *different and worse* failure mode than the original 421-on-replay — the same signature T24 already proved for `/search`: a cold, interaction-free navigation doesn't reliably fire these anti-bot-gated XHR calls. Two header sets, two distinct failure modes, three investigation rounds (T21/T22/T26) — this is "broken outright, not flaky," not a one-line fix, so per the dispatching brief's explicit fallback this session took **Outcome B** instead. - **Removed the whole `fetchAdDetails` opt-in.** Deleted from `.actor/input_schema.json`, `src/models.py`'s `ActorInput`, `README.md` (input table, example input/output, How-to-use step 5, FAQ entry), `promo.json` (sample input, differentiators, video hook), `.actor/actor.json`/`scaffold.json` descriptions, and `.actor/dataset_schema.json`. Dropped the 3 fields it populated (`external_url`/`call_to_action`/`advertising_objective`) from `src/models.py`'s `ResultRow`, `src/parser.py`'s `map_row` (and deleted `parser.merge_ad_details` entirely), and the dataset schema. Removed the implementation: `bootstrap.mint_details_token`, `bootstrap._capture_details_request`, `bootstrap.TokenBundle .details_headers`, `client.fetch_ad_details`, and `main.py`'s `_enrich_rows`/`_log_enrichment_outcome`/`_mint_details_for_page`. `_with_token_refresh` (REQ-11) is now `/search`-token-only — it lost its `ad_id` parameter and the details-mint branch of `_remint_after_expiry` (which is also deleted; the one remaining branch was folded inline). An Actor that ships fewer fields honestly is publishable; one that advertises 3 fields and returns null on every row is not. - **Item 1 (loud-failure gap on 100%-enrichment-failure) is now MOOT.** The dispatching brief flagged that `_log_enrichment_outcome` (from commit `c21e6b05`) degraded a total enrichment failure to a WARNING/ ERROR *log line* without failing the run or surfacing it in the dataset. Since the entire enrichment path no longer exists, there is nothing left to make loud — explicitly not skipped, just moot. - **Item 2 — pinned the proxy exit country (fixed, verified).** `bootstrap.py` launches Camoufox with a fixed `locale="en-US"`, but the shipped `proxyConfiguration` default carried no `apifyProxyCountry` — a geo-random `RESIDENTIAL` exit against a static locale is exactly the fingerprint mismatch `reference-pin-proxy-country` warns can park a session on an interstitial indefinitely. Added `"apifyProxyCountry": "US"` to the default/prefill in `.actor/input_schema.json`, `src/models.py`'s `ActorInput.proxy_configuration` default, and `tests/fixtures/input.qa.json` — matching, never `geoip=True` (that sweeps third-party IP-echo services through the proxy before launch and was the documented dominant driver of a chronic public-run-failure rate on `ai-overview-citations`). - **Cloud-measured query-level delivery rate on the final build, proxy-country-pinned, 4 runs / 9 queries, `region: "all"`, no overrides beyond the country pin:** | run id | queries | rows | completion | |---|---|---|---| | `p4MsGu3Mxl0UfyViR` | nike, zara, adidas (cap 15) | 45 | 3/3 | | `TDGwlVbvvGmwRFnss` | puma, reebok (cap 15) | 30 | 2/2 | | `0JcVZYQKUpMLQhWxd` | gucci, prada, chanel (cap 15) | 45 | 3/3 | | `bJWdvpbbqflogeRDV` | nike (cap 10, enrichment probe run) | 10 | 1/1 | **9/9 queries completed (100%)** — every run's status message reads `Done — N row(s) scraped across K/K quer(y/ies) completed.` with `K == len(queries)`. This is a small sample (one batch, ~3 minutes apart) and does not override the fleet's longer-run 87.5% figure from T24, but it shows no regression from the proxy-country pin and no repeat of "yesterday's" documented 3-of-4-runs-with-a-failing-query pattern in this batch. `chargedEventCounts` on all 4: `actor-start: 1` (fired once, on delivery), `ad-result` matching row count exactly — no over- or under-charging. - Verified locally: `ruff check` clean, `pyright` clean (0 errors on `src/`; the 4 pre-existing pyright errors in untouched test files — `test_bootstrap.py`, `test_models.py`, `test_parser.py` — predate this change, noted in 0.3's entry), full hermetic suite green (75 passed, 20 dead enrichment-only tests removed rather than left red), plus `scripts/verify_input_prefill.py`, `verify_no_scaffold_stub.py`, `verify_readme_price.py`, `verify_schema_refs.py`, `verify_actor_implemented.py`, `verify_actor_source_tracked.py`, `verify_charge_order.py`, `verify_ppe_sanity.py`, `verify_pydantic.py`, `verify_seo.py`, `verify_test_imports.py` — all OK. - **Recommendation: PUBLISH.** The one advertised-but-undeliverable feature is gone, the remaining default path measured 100% query completion across this session's 4 fresh cloud runs, PPE charges fire correctly and only on delivery, and a FAILED run bills $0.00 (verified 2026-09-18, unchanged by this diff). See `scripts/publish_actors.py`'s `SHELVED` dict — this session removed the `tiktok-ad-library-scraper` entry with a comment naming this CHANGELOG section as the retiring evidence. ### 0.3 (unpublished — fix on branch, not yet pushed) — 2026-09-18 — T25: enrichment fault isolation + zero-delivery exit gate - **Root cause, proven by a cloud run on build 0.2.4 (`O3wi2ORDDI4xZr7bE`, `fetchAdDetails: true`): one ad's enrichment failure killed the entire query and discarded every row the `/search` step had already found — 50 ads matched, 0 delivered.** The failure chain: a lazy per-page `/details` mint failed non-fatally (by design), the first enrichment call then 421'd, REQ-11's one-shot re-mint-and- retry itself raised `BootstrapError` (a `RuntimeError` subclass), and nothing inside `_enrich_rows`'s per-row loop caught it — it escaped all the way to `_run_all_queries`'s per-query handler, which correctly marked the query FAILED but by then `_run_query` had already thrown away the base rows it hadn't pushed yet (push happens after enrichment). This is the fleet's #1 documented failure class (`reference-fleet-fault-isolation-pattern`): a recoverable per-item error must never abort the whole run. - **Fix 1 — per-ad fault isolation (`src/main.py::_enrich_rows`).** Each row's enrichment attempt is now wrapped in its own `try/except RuntimeError`. A failure ships the base row unenriched (`external_url`/`call_to_action`/`advertising_objective` stay their `None` default) and the loop moves to the next ad — it never drops a row and never aborts the query. Enrichment successes/failures are counted per page; any failures log a `WARNING`, and a page where *every* enrichment failed logs an `ERROR` (a visible degradation signal) but still ships all its base rows — enrichment is optional extra fields on rows that already exist, so losing them for the fetchAdDetails=true customer over an optional-field failure would just be a slower way of committing the same bug. - **Fix 2 — zero-delivery fails loud (`src/main.py::_report_outcome`).** A run where not one single query ever completed (every outcome is the "failed after retries" sentinel) now raises `SystemExit(1)` instead of exiting 0 with an empty dataset and a green status (`reference-success-is-not-delivery`). A query that genuinely completed with zero matches (outcome `0`) is unaffected and keeps SUCCEEDING — the house "empty is not a failure" rule (`DEVILSCRAPES-CONTEXT.md` Anti-blocking-stack #6) is untouched. - **`_StartFee` behaviour is unchanged** — the start fee still only charges on the first batch that reaches the dataset, so this was always a correctness bug, not a billing one (the customer was billed $0 for the broken run). - The default path (`fetchAdDetails: false`) is untouched by this diff. Two spaced green cloud runs already exist on build 0.2.4 at the default prefill; this change does not invalidate them. - New regression tests in `tests/test_main.py`: a single failing enrichment still ships all base rows (the failed one unenriched, the others fully enriched); a page where every enrichment fails still ships all base rows and logs the degradation; `_report_outcome` fails loud only when zero queries ever completed, and keeps succeeding on a genuine zero-match or a mixed result. - Verified locally: `ruff check` clean, `pyright` clean on the touched files (`src/main.py`, `tests/test_main.py` — 4 pre-existing pyright errors in untouched test files are unrelated to this change), full hermetic suite green (94 passed, includes the 5 new T25 tests) plus the 2 `-m smoke` tests, `scripts/verify_input_prefill.py` and `scripts/verify_no_scaffold_stub.py` both OK. Not yet pushed — cloud QA is intentionally left to the human owner of this fix. ### 0.2 (build 0.2.3) — 2026-09-14 — T24: deterministic search-token bootstrap - **Root cause: T23's retry loop was still racy, not just under-retried.** A same-day measurement (build 0.2.2, the memory-OOM fix) found 3/4 spaced cloud runs SUCCEEDED on the shipped RESIDENTIAL/4096MB default, but one FAILED with the identical `BootstrapError: No /api/v1/search request observed during bootstrap` T23 was supposed to have fixed. Two structural races remained in `src/bootstrap.py`'s `_trigger_throwaway_search_with_retries`: (1) all 3 attempts drove the SAME page/DOM state, so a failed attempt could leave the page in a "search already submitted" state the next attempt's locators never matched; (2) success was decided by one fixed `wait_for_timeout(20_000)` then a single check, so a token arriving a few seconds late under proxy latency read as a hard failure. - **Two stronger fixes were tried first and ruled out with live evidence** (not assumed): calling `fetch()` directly from the page's own JS context (bypassing the UI) returned HTTP 421 "system busy" — the anti-bot token is computed inline by the click/submit handler, not injected by a global fetch/XHR patch a raw `fetch()` would also trigger. Navigating directly to the exact URL the SPA's own router writes to `history` after a real search never re-fires `/api/v1/search` on a fresh load — the request is tied to a live user-gesture event, not to page-load/route state. Both rule out a navigation-only or credential-harvesting fix. - **The fix**: `_mint_search_token` now gives every attempt a **fresh navigation** (no DOM state survives a failed attempt into the next one), and a new `_wait_for_search_token` decides success by **polling** the captured token in 500ms ticks up to a 25s bound, returning the instant it is observed instead of sleeping a fixed window regardless of when the token actually arrives. See `src/bootstrap.py`'s module docstring (T24) for the full detail. - Verified locally: full hermetic suite green (89 tests, including 2 new T24 regression tests — fresh navigation per attempt, and a token that only arrives after several poll ticks), `ruff`/`pyright` clean, and 3/3 real local Camoufox smoke runs against the live target passed. - **Cloud-measured, build 0.2.3, shipped RESIDENTIAL/4096MB default, no overrides, 8 spaced runs in ~10 minutes: 7/8 SUCCEEDED (87.5%)**, up from 3/4 (75%) on build 0.2.2 (the pre-T24 code). Run IDs and full analysis in `docs/specs/tiktok-ad-library-scraper/notes.md`'s T24 entry. **Not fully deterministic**: one run (`m5qsrRwsjdutfQRjQ`) still FAILED with the identical error after exhausting all 3 fresh-navigation attempts, and — unlike every prior failure this Actor has logged — every UI interaction step (cookie dismiss, region select, click, type, submit) completed with **zero per-step warnings**, meaning the click path itself worked mechanically every time; the token simply never appeared. That points at target-side anti-bot session variance (a browser-fingerprint/IP combination the risk engine declines to trust for that run), not a client-side timing race — see notes.md for the full reasoning. Still `isPublic: false` per explicit instruction; this fix measurably reduces but does not eliminate the failure mode, so it stays shelved pending a human call on whether 87.5% is acceptable. ### 0.2 — 2026-09-13 (unpublished — fix on branch, not yet pushed) - **Bound-and-retry the search-trigger click (T23)**: cloud QA on the shipped RESIDENTIAL default FAILED with "No /api/v1/search request observed during bootstrap", one line below "Throwaway search trigger failed: Locator.click: Timeout 5000ms exceeded". The entire token mint depended on one 5-second locator click succeeding, with no retry and no alternative path — a single slow render sank the whole run. A prior session initially misdiagnosed this as a proxy-tier effect (datacenter FAIL, residential PASS) before a second residential run on the same tier also failed, ruling that out. `src/bootstrap.py` now retries the search trigger up to `SEARCH_TRIGGER_MAX_ATTEMPTS=3` times with a longer per-attempt timeout (`SEARCH_TRIGGER_TIMEOUT_MS=10s`, up from 5s), and adds an independent submit path — pressing Enter in the search field — if the Search button click itself fails. Still raises `BootstrapError` (fail loud) once every attempt is exhausted; bounding the retry does not turn a real block into a silent empty success. Proxy tier is unchanged (still RESIDENTIAL, the existing shipped default) — this fix targets the actual root cause instead of the tier. Verified locally: full hermetic suite green (89 tests, including 4 new T23 regression tests covering the retry/fallback/fail-loud paths), `ruff`/`pyright` clean, and the live `-m smoke` Camoufox tests pass twice against the real target. **Not yet pushed to Apify** — see `docs/specs/tiktok-ad-library-scraper/notes.md` for why a cloud QA run wasn't fired in this pass. ### 0.1 (same-day correction) — 2026-09-12 - **Critical transport correction, found during pipeline-manager verification, not by the original build**: plain `curl-cffi` alone cannot reach `POST /api/v1/search` or `GET /api/v1/items//details` — both require a browser-minted `x-ccl-str` anti-bot header. The 0.1.0 build below shipped believing otherwise (its own fixtures were built from recon prose, not a fresh capture, after live probing during that build session got a misleading uniform `421`). Direct isolation proved it's not IP-level throttling (a fresh Camoufox call from the same IP in the same minute got `200`) and that the token is reusable, not single-request-bound. Fix: `src/bootstrap.py` (new) does one Camoufox visit per run to mint the `/search` token (survives 30s+, reused for the whole run's search + pagination traffic via `curl-cffi impersonate="firefox133"`), plus a lazy, per-page `/details` token (much shorter TTL, ~single-digit seconds) minted right before that page's enrichment loop rather than once at boot. `client.py` gained `TokenExpiredError` (HTTP 421) as a distinct retry path from REQ-7's 429/503 backoff — one re-mint-and-retry, then fail that query loud (REQ-11). `.actor/Dockerfile` now downloads the Camoufox browser (`camoufox fetch`, above `COPY src`); `pyproject.toml` adds `camoufox[geoip]`. See `research/RECON-tiktok-ad-library-2026-09-12.md`'s "CORRECTION" section and `docs/specs/tiktok-ad-library-scraper/notes.md` for the full live-target isolation. - **`fetchAdDetails` now defaults to `false` (was `true`)**, after a live measurement: even with the lazy per-page mint fix above, a full `maxResultsPerQuery=25` × 2-query enrichment run needed ~24 token re-mints for 50 rows (~1 every 2 rows, each costing a real Camoufox relaunch) — too slow/costly to be the default. Search + pagination + all list-level fields (advertiser, caption, dates, estimated audience, creative URLs) are unaffected and fast (no re-mints at all in a clean `maxResultsPerQuery=25`/`fetchAdDetails=false` run, ~40s wall-clock for 50 rows across 2 queries). Detail enrichment (`external_url`/`call_to_action`/`advertising_objective`) remains available as an explicit opt-in, documented in the README with a latency caveat. - This Actor now genuinely drives Camoufox, so the fleet's structural anti-bot-class publish gate (`scripts/publish_actors.py`'s `_is_antibot_class`) will correctly classify it as anti-bot class — two green cloud QA runs ≥1h apart are required before publish, not one. ### 0.1.0 — 2026-09-12 - Real crawler implemented: `src/client.py` (curl-cffi transport, exponential backoff on `{429, 503}` + network errors per REQ-7, `fetch_support_regions`, `search_page`, `fetch_ad_details`), `src/parser.py` (pure `build_search_body`/`build_time_window`/ `epoch_ms_to_iso`/`map_row`/`merge_ad_details`), `src/main.py` (per-query cursor-pagination loop, REQ-2 live region gate, REQ-6 zero-rows-is-success handling, REQ-10 per-query fault isolation, REQ-5 optional ad-detail enrichment, PPE charging, proxy resolution). - `src/models.py`'s `ResultRow` filled in with the full ad-row shape (`ad_id`, `advertiser_name`, `video_url`, `estimated_audience`, `external_url`, etc.) per design.md's Output Schema. - `.actor/pay_per_event.json` added (`actor-start` $0.20 flat, `ad-result` $0.003/row — ~$3.20/1,000 ads). `.actor/dataset_schema.json` filled in with the real field set + nullable typing + an `overview` view. `actor.json`/`pyproject.toml` bumped to 0.1. - `promo.json` + `scaffold.json` added; README rewritten with the real Store-listing copy (10 canonical H2 sections, price banner, input/ output tables, EU-only-coverage limitation stated plainly). - **Live-verified correction vs. design.md**: `GET /api/v1/support-regions` returns `{"regions": [{"code": "AT", "name": "Austria"}, ...], "code": 0}`, not the `{"data": []}` shape design.md assumed. `client.fetch_support_regions` extracts `code` from each `{code, name}` object; the flat `list[str]` of codes it returns to callers is unchanged, so the correction stayed local to that one function. See `tests/fixtures/support_regions_response.json` — a fresh live capture taken during this build (2026-09-12), 33 region codes (matches spec.md's Assumptions note that the recon's own "34 regions" prose and 33-code enumeration already disagreed). - `tests/fixtures/search_response_page1.json` / `search_response_page2.json` / `item_details_response.json` are built from `research/RECON-tiktok-ad-library-2026-09-12.md`'s captured row shape and example values (not a fresh live capture) — repeated `/search` and `/items/.../details` probes during this build session returned a consistent `421 "system busy"` regardless of region/body variation, including through an Apify residential proxy pinned to `country-DE`, most consistent with a soft rate-limit triggered by the probing itself rather than a genuine wire-contract change (`support-regions`, a bare GET, kept returning 200 throughout). Flagged for `actor-qa-engineer` to confirm the `/search`/`/items/.../details` wire shapes with a single, paced cloud run rather than repeated manual probing. - 58 tests passing (`test_models.py`, `test_parser.py`, `test_client.py`, `test_main.py`, `test_charge_no_idempotency_key.py`), ruff clean, pyright clean (`uv run pyright`). ### 0.0.1 — 2026-09-12 - Scaffolded: `.actor/` (`actor.json` version `0.0`, `categories: ["MARKETING", "LEAD_GENERATION"]`, `seoTitle` / `seoDescription` set, `input_schema.json` matches design.md's full `ActorInput` shape — `queries`, `region`, `dateFrom`/`dateTo`, `maxResultsPerQuery`, `fetchAdDetails`, `proxyConfiguration` — `dataset_schema.json` / `output_schema.json` placeholder stubs, `Dockerfile` with `CMD ["python", "-m", "src"]`), `src/main.py` wiring (reads input, pushes one clearly-marked placeholder record via `{"placeholder": true, ...}`, exits clean), `src/__init__.py` / `src/__main__.py` (bare `asyncio.run(main())`, no guard needed under `python -m src`), `src/models.py` with a full `ActorInput` (matching design.md's date-window/query-length validators) and a placeholder `ResultRow`, test scaffold (`tests/test_main.py`), QA fixture (`tests/fixtures/input.qa.json`, deliberately >1 page per query per design.md's Test Strategy), README skeleton. - Proxy default is plain `{"useApifyProxy": true}` with no `apifyProxyGroups` key, per REQ-9 — no `RESIDENTIAL` literal appears anywhere in this Actor's tree; recon showed zero blocks. - `.actor/icon.svg` deferred to `actor-icon-designer` (TikTok brand mark inside the canonical Devil Scrapes container). - Real crawler logic (`src/client.py`, `src/parser.py`, the real `src/main.py` cursor-pagination loop, `src/models.py`'s `ResultRow`, `.actor/pay_per_event.json`, `promo.json`) is not implemented yet — see `docs/specs/tiktok-ad-library-scraper/tasks.md`.