Ghost Fetch
Under maintenancePricing
Pay per usage
max_nav_ms (default 45000) caps the page.goto time-box; the box actually used
is derived per attempt from that attempt's own remaining time-box (max_browser_attempt_ms clipped
to the remaining max_total_ms), capped by max_nav_ms and floored at the old constant, 15000. An
explicit budget.max_nav_ms is the cap, so max_nav_ms: 15000 pins the old behaviour. Before this,
every navigation got a hard-coded 15 s regardless of how much budget the attempt held: OBSERVED on the
deployed standby (forced browser, alza.cz, after #42 raised the browser-initial attempt box to 120 s),
first attempts failed as transport_error with TimeoutError: page.goto: Timeout 15000ms exceeded
after having already spent 33–60 s on gateway allocation — the navigation was cut with ~60 s of its own
attempt budget unspent, and each such failure cost a proxy rotation and a second gateway allocation.max_nav_ms 15 000 vs 45 000. Both arms: 8/8 success. The 45 000 arm
spent 8 browser attempts, mean cost 12.0, median latency 53.7 s; the 15 000 arm spent 9 attempts, mean
cost 13.5, median latency 56.5 s — because one attempt hit the box at 15 005 ms, was reported as a
transport_error, rotated the exit IP and paid a second gateway allocation (cost 24, 102 s). 4 of 8
navigations in the 45 000 arm needed 15.5–17.5 s to reach domcontentloaded, i.e. this target's
honest time to first document sits right on top of the old 15 s constant. No run in either arm ran out
of its attempt budget (longest 102 s of a 120 s box), so the time the 15 s box refused to spend was
always there.decision_trace[].timing with lease_ms
(gateway allocate + CDP connect + window acquisition, reported even when the lease FAILED),
pre_nav_ms (warmup nav + dispatch), nav_ms, nav_timeout_ms and nav_timed_out. Without the split
a 15 s box cutting a navigation is indistinguishable from a slow target. Measured share on the same
A/B: lease_ms median ~12.5 s (~22–25 % of a ~50 s attempt), pre_nav_ms bimodal at ~13 s or ~43 s,
nav_ms 11.2–17.5 s.max_nav_ms boxes navigations only.docs/api-v1-fetch.md (Budget table), docs/trace-v4.md (Browser Attempt Timing),
docs/operations.md (read timing before blaming the target).max_total_ms / max_browser_attempt_ms). A request that starts on the browser
(render.mode: "browser", a screenshot, or a browser-only gateway deploy where the HTTP tier is not
dialable) pays gateway allocation before its first byte, so a Cloudflare-fronted nav does not
reliably fit in 60 s: forced-browser fetches of alza.cz ended at 60.0 s as budget_exhausted /
status: 0 / 12 cost units on a single attempt, while the same fetch with max_total_ms: 170000
succeeded in 55 s. The browser cap moves WITH the total, so a browser-initial run buys one long
attempt. Measured on the deployed gateway (forced browser, 3 runs) with a 60 s cap under a 120 s
total: one run succeeded in 46 s, two hit the cap, rotated the exit IP and were cut again at ~60 s —
budget_exhausted at 120 000 ms, cost 24, two browser allocations; single attempts capped at
120 000 ms succeeded in 55 s and 92 s. The second attempt only doubled the cost, and an attempt that
does blow the budget now reports the honest budget_exhausted. Explicit caller values always win —
set max_browser_attempt_ms below max_total_ms to keep the recoverable rotate-and-retry path.gateway 502 exit_verification_failed on the
browser allocation) and the rotated retry then hit the clock, the verdict stays budget_exhausted
but the envelope carries gateway.backend_failure and the give_up reason reads
budget exhausted after gateway allocation failure: …. Before, the client saw only
budget_exhausted and could not tell a gateway outage from a slow site.docs/api-v1-fetch.md: the Budget table records both tier-dependent defaults, and the verdict
guidance tells callers to check gateway.backend_failure before raising the budget.max_total_ms / max_attempts / max_cost_units cuts a
run off mid-attempt, the envelope now reports verdict: "budget_exhausted" (previously it
reported the previous completed attempt's verdict — e.g. soft_block — so a 60 s budget
blow-out was indistinguishable from a real block). response / final_signal still carry the
last completed attempt as evidence.min_browser_attempt_ms (default 20 000). The Recovery Policy no longer
starts a target browser attempt when less than this remains of max_total_ms; it takes the
proxy-rotation + HTTP replay path instead. Before, http(403) → warmup → browser started the
browser nav with ~20–30 s left, could not finish, burned the remainder plus a gateway browser
allocation, and returned at exactly max_total_ms.docs/api-v1-fetch.md gains a per-knob Budget table and verdict guidance for
budget_exhausted vs soft_block.ghost-fetch now keeps only the backbone — resolve Context → pure Recovery Policy ladder → gateway transports → Verifier Verdict → envelope. Three systems were removed whole:
src/strategy-learning.ts): Thompson sampling, the per-arm Beta
posterior, reward and label machinery. Selection is now a deterministic ladder — HTTP first when
dialable, escalate to the browser tier otherwise. Supersedes ADR-0002.src/recovery-playbooks.ts): the two profile-evidence rules
(skipFailedClearanceReplay, rotateAfterRepeatedFailedBrowserRecovery). Every branch they
overrode now takes its baseline action. The pure decide() seam and its action alphabet are
unchanged (ADR-0001).src/profiles.ts): the HostContextProfileV1 materializer, the
profile KV store, the profiles:* scripts, and GHOST_FETCH_PROFILE_KV_STORE_ID. Supersedes
ADR-0003.No behaviour change given an empty profile store: the bandit selector and the playbook rules
were both live code on every fetch (playbooks default-on, selector default upgrade_only), but
inert without a materialized profile for the fetched context — no profile → the selector hits
empty_posterior and the playbook gates fail, so both fall through to the deterministic ladder. The
store was populated only by the offline profiles:* scripts; an unseeded deployment always took the
ladder. A deployment that had seeded profiles could see a different action sequence — the same
reachable verdict in the common case, though at a max_attempts/max_cost_units boundary a skipped
attempt could flip the outcome — so the removal is safe on the empty-store basis rather than on any
config flag.
Backward compatible. policy.strategy_selection is still accepted and is now a deprecated
no-op (ignored). The envelope still carries strategy_learning with its full pre-refactor field
set — mode, selected_initial_strategy, actual_initial_strategy, selection_reason, gate,
reward_direct, reward_chain, posterior_snapshot — filled with honest no-bandit values (empty
gate, rewards 0, empty posterior), so a v1 consumer's strict schema still validates. The shape is
compatible; the values are not identical to the old upgrade_only default (a consumer that branched
on mode/posterior_snapshot values rather than reading the fields sees a change).
TRACE_SCHEMA_VERSION is unchanged at 4. Telemetry remains as off-by-default observability, now
with nothing consuming it.
country semantics changesrc/context.ts owns the Context tuple and its 5-part key
(host|country|method|path_group|proxy_group); the orchestrator resolves it once
per fetch and threads it into selection, recovery playbooks, and telemetry.country: "ANY" (or PROXY_COUNTRY) — the egress actually used — instead of the
invented "US" default. Historical v1 rows with country: "US" are ambiguous
(explicit US vs the old default). Structurally compatible; no schema-version bump.host|us|… while selection and playbooks read host|any|….host: "unknown" instead of "".GHOST_FETCH_TELEMETRY_RAW=1 attaches the full raw fetch result — the same payload /v1/fetch returns, including the HTML body, response headers, final_url, decision trace, and base64 screenshot — to the request telemetry event under a raw_response field. Off by default; sanitized telemetry is unchanged (backward compatible).GHOST_FETCH_TELEMETRY_RAW and a valid custom GHOST_FETCH_TELEMETRY_DATASET_ID, so raw payloads (bodies, set-cookie headers, full URLs) never reach the shared default ghost-fetch-telemetry dataset that feeds Passive Context Profiles.GHOST_FETCH_TELEMETRY_DATASET_ID is now validated as an Apify Dataset id (17-char alphanumeric) or name (name / username/name); a malformed value warns and falls back to the default dataset instead of wedging writes. The dataset open is also no longer cached on failure.body is bounded by GHOST_FETCH_TELEMETRY_RAW_MAX_BYTES (default 6 MB, fits Apify's ~9 MB item limit); when truncated the event carries raw_response_truncated: true.pip install "cloverlabs-camoufox[geoip]" + python3 -m camoufox fetch official/150.0.2-alpha.26.rt:c captcha. cloverlabs FF150 → auto-solvable rt:i interstitial.fox_main_world_eval request paramcloverlabs/FF150 has no Juggler-scope isolation (single-world model). The flag set camoufox's allowMainWorld:True which is now inert; the mw: prefix it was meant to enable is parsed as a JavaScript label. The param has no observable effect on this build and is dropped from the request schema and the FoxOptions interface. See proxy-mcp v3.3.0 for the underlying world-model investigation and scripts/camoufox-world-probe.ts to verify on any installed build.
inPageFetch no longer hidden from page hooksinPageFetch() (src/fetch.ts:498) issues fetch() via page.evaluate to avoid Akamai poisoning of GET navigation. Under daijro/FF135 the eval ran in a Juggler scope, isolated from any page-installed window.fetch hooks. Under cloverlabs/FF150 it runs in the page's main world — pages that have hooked window.fetch (Akamai, PerimeterX) now observe the request. No correctness change; potential detection signal on high-tier WAFs.