# Changelog of Ghost Fetch (`straightforward_understanding/ghost-fetch`) Actor - **URL**: https://apify.com/straightforward_understanding/ghost-fetch/changelog.md - **Full Actor documentation**: https://apify.com/straightforward_understanding/ghost-fetch.md ## Changelog ### Unreleased #### Fix: the browser navigation time-box comes from the attempt budget, not a constant (#43) - **New budget field `max_nav_ms` (default `45000`) caps the `page.goto` time-box; the box actually used is derived per attempt** from that attempt's own remaining time-box (`max_browser_attempt_ms` clipped to the remaining `max_total_ms`), capped by `max_nav_ms` and floored at the old constant, `15000`. An explicit `budget.max_nav_ms` is the cap, so `max_nav_ms: 15000` pins the old behaviour. Before this, every navigation got a hard-coded 15 s regardless of how much budget the attempt held: OBSERVED on the deployed standby (forced browser, alza.cz, after #42 raised the browser-initial attempt box to 120 s), first attempts failed as `transport_error` with `TimeoutError: page.goto: Timeout 15000ms exceeded` after having already spent 33–60 s on gateway allocation — the navigation was cut with ~60 s of its own attempt budget unspent, and each such failure cost a proxy rotation and a second gateway allocation. - **Live A/B decided the default.** Local gateway mode, forced browser, alza.cz PDP, n = 8 per arm, sequential and interleaved, `max_nav_ms` 15 000 vs 45 000. Both arms: 8/8 `success`. The 45 000 arm spent 8 browser attempts, mean cost 12.0, median latency 53.7 s; the 15 000 arm spent 9 attempts, mean cost 13.5, median latency 56.5 s — because one attempt hit the box at 15 005 ms, was reported as a `transport_error`, rotated the exit IP and paid a second gateway allocation (cost 24, 102 s). **4 of 8 navigations in the 45 000 arm needed 15.5–17.5 s** to reach `domcontentloaded`, i.e. this target's honest time to first document sits right on top of the old 15 s constant. No run in either arm ran out of its *attempt* budget (longest 102 s of a 120 s box), so the time the 15 s box refused to spend was always there. - **A browser attempt now reports where its wall time went** — `decision_trace[].timing` with `lease_ms` (gateway allocate + CDP connect + window acquisition, reported even when the lease FAILED), `pre_nav_ms` (warmup nav + dispatch), `nav_ms`, `nav_timeout_ms` and `nav_timed_out`. Without the split a 15 s box cutting a navigation is indistinguishable from a slow target. Measured share on the same A/B: `lease_ms` median ~12.5 s (~22–25 % of a ~50 s attempt), `pre_nav_ms` bimodal at ~13 s or ~43 s, `nav_ms` 11.2–17.5 s. - Unchanged on purpose: the 3 s post-4xx challenge-resolution short path, the challenge-resolution ceilings, and the render-readiness wait. `max_nav_ms` boxes navigations only. - `docs/api-v1-fetch.md` (Budget table), `docs/trace-v4.md` (Browser Attempt Timing), `docs/operations.md` (read `timing` before blaming the target). #### Fix: a browser-initial request gets a budget sized for the browser tier (#42) - **Budget defaults now depend on the initial tier: 60 000 / 60 000 HTTP-initial, 120 000 / 120 000 browser-initial** (`max_total_ms` / `max_browser_attempt_ms`). A request that starts on the browser (`render.mode: "browser"`, a screenshot, or a browser-only gateway deploy where the HTTP tier is not dialable) pays gateway allocation before its first byte, so a Cloudflare-fronted nav does not reliably fit in 60 s: forced-browser fetches of alza.cz ended at 60.0 s as `budget_exhausted` / `status: 0` / 12 cost units on a single attempt, while the same fetch with `max_total_ms: 170000` succeeded in 55 s. The browser cap moves WITH the total, so a browser-initial run buys one long attempt. Measured on the deployed gateway (forced browser, 3 runs) with a 60 s cap under a 120 s total: one run succeeded in 46 s, two hit the cap, rotated the exit IP and were cut again at ~60 s — `budget_exhausted` at 120 000 ms, cost 24, two browser allocations; single attempts capped at 120 000 ms succeeded in 55 s and 92 s. The second attempt only doubled the cost, and an attempt that does blow the budget now reports the honest `budget_exhausted`. Explicit caller values always win — set `max_browser_attempt_ms` below `max_total_ms` to keep the recoverable rotate-and-retry path. - **A gateway backend failure is no longer hidden by a later budget cut.** When the run's last completed attempt failed inside the gateway (e.g. `gateway 502 exit_verification_failed` on the browser allocation) and the rotated retry then hit the clock, the verdict stays `budget_exhausted` but the envelope carries `gateway.backend_failure` and the give_up reason reads `budget exhausted after gateway allocation failure: …`. Before, the client saw only `budget_exhausted` and could not tell a gateway outage from a slow site. - `docs/api-v1-fetch.md`: the Budget table records both tier-dependent defaults, and the verdict guidance tells callers to check `gateway.backend_failure` before raising the budget. #### Fix: a browser attempt that cannot fit the remaining budget no longer pins the fetch (#41) - **Honest verdict on exhaustion.** When `max_total_ms` / `max_attempts` / `max_cost_units` cuts a run off mid-attempt, the envelope now reports `verdict: "budget_exhausted"` (previously it reported the *previous* completed attempt's verdict — e.g. `soft_block` — so a 60 s budget blow-out was indistinguishable from a real block). `response` / `final_signal` still carry the last completed attempt as evidence. - **New budget knob `min_browser_attempt_ms` (default 20 000).** The Recovery Policy no longer starts a target browser attempt when less than this remains of `max_total_ms`; it takes the proxy-rotation + HTTP replay path instead. Before, `http(403) → warmup → browser` started the browser nav with ~20–30 s left, could not finish, burned the remainder plus a gateway browser allocation, and returned at exactly `max_total_ms`. - `docs/api-v1-fetch.md` gains a per-knob **Budget** table and verdict guidance for `budget_exhausted` vs `soft_block`. #### Backbone: the learning stack is removed (bandit + playbooks + passive profiles) ghost-fetch now keeps only the backbone — resolve Context → **pure Recovery Policy ladder** → gateway transports → Verifier Verdict → envelope. Three systems were removed whole: - **Strategy-selection bandit** (`src/strategy-learning.ts`): Thompson sampling, the per-arm Beta posterior, reward and label machinery. Selection is now a deterministic ladder — HTTP first when dialable, escalate to the browser tier otherwise. Supersedes [ADR-0002]. - **Recovery playbooks** (`src/recovery-playbooks.ts`): the two profile-evidence rules (`skipFailedClearanceReplay`, `rotateAfterRepeatedFailedBrowserRecovery`). Every branch they overrode now takes its baseline action. The pure `decide()` seam and its action alphabet are unchanged ([ADR-0001]). - **Passive Context Profiles** (`src/profiles.ts`): the `HostContextProfileV1` materializer, the profile KV store, the `profiles:*` scripts, and `GHOST_FETCH_PROFILE_KV_STORE_ID`. Supersedes [ADR-0003]. **No behaviour change given an empty profile store:** the bandit selector and the playbook rules were both live code on every fetch (playbooks default-on, selector default `upgrade_only`), but inert without a materialized profile for the fetched context — no profile → the selector hits `empty_posterior` and the playbook gates fail, so both fall through to the deterministic ladder. The store was populated only by the offline `profiles:*` scripts; an unseeded deployment always took the ladder. A deployment that *had* seeded profiles could see a different action sequence — the same reachable verdict in the common case, though at a `max_attempts`/`max_cost_units` boundary a skipped attempt could flip the outcome — so the removal is safe on the empty-store basis rather than on any config flag. **Backward compatible.** `policy.strategy_selection` is still accepted and is now a deprecated no-op (ignored). The envelope still carries `strategy_learning` with its **full pre-refactor field set** — `mode`, `selected_initial_strategy`, `actual_initial_strategy`, `selection_reason`, `gate`, `reward_direct`, `reward_chain`, `posterior_snapshot` — filled with honest no-bandit values (empty gate, rewards `0`, empty posterior), so a v1 consumer's strict schema still validates. The shape is compatible; the values are not identical to the old `upgrade_only` default (a consumer that branched on `mode`/`posterior_snapshot` values rather than reading the fields sees a change). `TRACE_SCHEMA_VERSION` is unchanged at `4`. Telemetry remains as off-by-default observability, now with nothing consuming it. [ADR-0001]: docs/adr/0001-recovery-policy-decision-seam.md [ADR-0002]: docs/adr/0002-bandit-reward-and-rollout.md [ADR-0003]: docs/adr/0003-durable-learning-via-telemetry-profiles.md #### One shared Context key (ADR-0004) — telemetry `country` semantics change - New `src/context.ts` owns the Context tuple and its 5-part key (`host|country|method|path_group|proxy_group`); the orchestrator resolves it once per fetch and threads it into selection, recovery playbooks, and telemetry. - **Behavior note for telemetry consumers:** country-less request events now record `country: "ANY"` (or `PROXY_COUNTRY`) — the egress actually used — instead of the invented `"US"` default. Historical v1 rows with `country: "US"` are ambiguous (explicit US vs the old default). Structurally compatible; no schema-version bump. - Fixes the silent learning miss where a country-less fetch wrote profile evidence under `host|us|…` while selection and playbooks read `host|any|…`. - Malformed request URLs now key (and record) `host: "unknown"` instead of `""`. ### 0.3.1 — 2026-06-10 #### Opt-in raw response capture in telemetry - New `GHOST_FETCH_TELEMETRY_RAW=1` attaches the full raw fetch result — the same payload `/v1/fetch` returns, including the HTML `body`, response headers, `final_url`, decision trace, and base64 screenshot — to the request telemetry event under a `raw_response` field. Off by default; sanitized telemetry is unchanged (backward compatible). - Double-gated: raw capture requires `GHOST_FETCH_TELEMETRY_RAW` **and** a valid custom `GHOST_FETCH_TELEMETRY_DATASET_ID`, so raw payloads (bodies, `set-cookie` headers, full URLs) never reach the shared default `ghost-fetch-telemetry` dataset that feeds Passive Context Profiles. - `GHOST_FETCH_TELEMETRY_DATASET_ID` is now validated as an Apify Dataset id (17-char alphanumeric) or name (`name` / `username/name`); a malformed value warns and falls back to the default dataset instead of wedging writes. The dataset open is also no longer cached on failure. - Raw `body` is bounded by `GHOST_FETCH_TELEMETRY_RAW_MAX_BYTES` (default 6 MB, fits Apify's ~9 MB item limit); when truncated the event carries `raw_response_truncated: true`. - See [docs/telemetry.md → Raw Response Capture](docs/telemetry.md#raw-response-capture). ### 0.3.0 — 2026-05-16 #### Camoufox dep swap: cloverlabs-camoufox 0.6.0 + Firefox 150 - Dockerfile: `pip install "cloverlabs-camoufox[geoip]"` + `python3 -m camoufox fetch official/150.0.2-alpha.26`. - Motivation: daijro/camoufox PyPI stale at Firefox 135 → DataDome serves unsolvable `rt:c` captcha. cloverlabs FF150 → auto-solvable `rt:i` interstitial. - README local-install snippet updated to match. #### Breaking — removed `fox_main_world_eval` request param cloverlabs/FF150 has no Juggler-scope isolation (single-world model). The flag set camoufox's `allowMainWorld:True` which is now inert; the `mw:` prefix it was meant to enable is parsed as a JavaScript label. The param has no observable effect on this build and is dropped from the request schema and the `FoxOptions` interface. See `proxy-mcp` v3.3.0 for the underlying world-model investigation and `scripts/camoufox-world-probe.ts` to verify on any installed build. #### Behavior note — `inPageFetch` no longer hidden from page hooks `inPageFetch()` (`src/fetch.ts:498`) issues `fetch()` via `page.evaluate` to avoid Akamai poisoning of GET navigation. Under daijro/FF135 the eval ran in a Juggler scope, isolated from any page-installed `window.fetch` hooks. Under cloverlabs/FF150 it runs in the page's main world — pages that have hooked `window.fetch` (Akamai, PerimeterX) now observe the request. No correctness change; potential detection signal on high-tier WAFs.