ghost-fetch

ghost-fetch is a standby Apify Actor that exposes one HTTP endpoint: POST /v1/fetch.

It fetches a single URL and returns the raw response body. It does not parse the page, crawl links, render Markdown, or write per-request results to a dataset.

Quickstart

Run locally:

npm install
APIFY_META_ORIGIN=STANDBY npm run start:dev

Fetch a clean URL:

curl -X POST http://localhost:3001/v1/fetch \
  -H "content-type: application/json" \
  -d '{"url":"https://example.com"}'

Hosted standby endpoint:

curl -X POST https://<user>--ghost-fetch.apify.actor/v1/fetch \
  -H "authorization: Bearer $APIFY_TOKEN" \
  -H "content-type: application/json" \
  -d '{"url":"https://example.com","country":"US"}'

Minimal response shape:

{
    "trace_schema_version": 4,
    "verdict": "success",
    "usable_content": true,
    "response": {
        "status": 200,
        "body": "<raw fetched payload>",
        "final_url": "https://example.com"
    },
    "final_signal": { "kind": "success", "confidence": 0.8 },
    "decision_trace": [],
    "strategy_learning": {
        "mode": "shadow",
        "selection_reason": "shadow_only"
    }
}

Versioning Note

Two version numbers exist and they move independently:

• API route version (v1): the URL /v1/fetch and its public request and response contract. This bumps only for breaking endpoint contract changes.
• Trace schema version (4): the shape of trace fields inside the /v1/fetch response, such as decision_trace, verification, and strategy_learning.

You can call /v1/fetch and receive "trace_schema_version": 4. That is normal. There is no /v4/fetch endpoint.

What It Does

ghost-fetch tries a cheap TLS-impersonating HTTP client first. If the response looks blocked, it can fall back to a stealth browser:

• Tier 1: impit HTTP impersonation, using a Chrome-like TLS profile.
• Tier 2: the ghost patched Chromium stealth browser.
• Passive WAF classifier: turns status, headers, cookies, body markers, URLs, and script hosts into typed signals.
• Bayesian semantic verifier: decides whether the returned body is usable content, including whether a 2xx body is only a thin/app-shell/incomplete page.
• Browser readiness: browser attempts wait on DOM/content-quality growth and expose a compact readiness summary in the trace.
• Strategy learner: records and, when explicitly enabled, can choose the first strategy after conservative gates pass.

The ghost patched Chromium is the only browser engine. render.browser accepts only chrome; a firefox request is rejected (the firefox/camoufox tier was removed — see ADR-0008).

Recovery after failures remains deterministic in the current phase.

The Actor works well as a hosted fetch primitive for agents, internal tools, and scraping pipelines that need one URL at a time. On Apify, it can run behind standby HTTP, API authentication, scheduling, monitoring, and residential proxy routing.

Why Use ghost-fetch?

• It exposes a simple HTTP API instead of a browser automation API.
• It returns the fetched payload as-is, so callers decide how to parse it.
• It avoids browser cost when Tier 1 succeeds.
• It supports country and session hints for proxy-backed targets.

Input

Request body:

{
    "url": "https://example.com",
    "country": "US",
    "session": "crawl-1",
    "method": "GET",
    "headers": { "accept-language": "en-US,en;q=0.9" },
    "body": "optional raw request body",
    "render": {
        "mode": "auto",
        "browser": "chrome",
        "screenshot": "none"
    }
}

Main fields:

Field	Type	Default	Notes
url	URL string	required	Target URL to fetch.
country	ISO-2 string	none	Residential proxy country when proxy credentials are configured.
session	string	auto-derived	Sticky cookie/proxy session key. When omitted, ghost-fetch derives a stable per-origin token from country + URL origin so cookie hydration works on follow-up calls without the caller having to mint one. Pass your own to isolate jars across crawlers.
method	GET, POST, PUT, PATCH, DELETE	GET	Non-GET requests are supported without screenshots.
headers	object	none	Extra request headers (Tier 1; and Tier 2 for non-GET via in-page fetch).
body	string	none	Raw request body for non-GET requests.
render	object	none	Optional browser/render controls (all fields optional).
render.mode	auto, http, browser	auto	http = Tier 1 only (no browser fallback); browser = force the browser tier; auto = full cascade.
render.browser	chrome	chrome	Browser engine: chrome = ghost patched Chromium (the only browser). A firefox request is rejected (ADR-0008).
render.screenshot	none, viewport, full_page	none	Screenshots force browser navigation and only work with GET.

Output

Response body (trace_schema_version: 4):

{
    "trace_schema_version": 4,
    "request_id": "…",
    "verdict": "success",
    "usable_content": true,
    "response": {
        "status": 200,
        "headers": { "content-type": "text/html" },
        "body": "<raw fetched payload>",
        "final_url": "https://www.example.com/"
    },
    "final_signal": { "kind": "success", "confidence": 1 },
    "decision_trace": [
        { "attempt": 1, "strategy_id": "impit_chrome", "transport": "http", "outcome": "success", "latency_ms": 842, "cost_units": 1 }
    ],
    "cost": { "attempt_count": 1, "cost_units": 1, "latency_ms": 842, "browser_latency_ms": 0, "rotation_count": 0 },
    "session": {
        "original_session": "auto15c99f857ba0",
        "effective_session": "auto15c99f857ba0",
        "rotation_count": 0,
        "diagnostics": {
            "origin_key": "FR:auto15c99f857ba0:https://www.example.com",
            "quality_score": 30,
            "consecutive_failures": 0,
            "suspect": false,
            "suspected_rate_limited": false,
            "last_failure_status": null,
            "last_failure_markers": []
        }
    }
}

The fetched payload is response.body (HTML stays HTML, JSON stays JSON text) — challenge pages are returned too, so check verdict (success vs soft_block / hard_block / verifier_failed / transport_error / budget_exhausted) and final_signal.kind before trusting the body. The tier that won and the browser family it dispatched against are in decision_trace (one entry per attempt, each with strategy_id, transport, browser, outcome). verification, strategy_learning, and response.screenshot (base64 PNG) are present only when they apply.

Session diagnostics

When a session is in play (either explicitly passed by the caller or auto-derived from country + URL origin), the response's session object (original_session, effective_session, rotation_count) carries a nested session.diagnostics object summarising the running quality of that (country, session, origin) tuple:

Field	Notes
origin_key	Internal key country:session:origin — handy when correlating multiple URLs that share a jar.
quality_score	Running tally. +10 per success, -10 per failure, clamped to [-100, 100]. Starts at 0.
consecutive_failures	Reset to 0 on success; +1 per failure.
suspect	Sticky on any non-rate-limit failure. Generic — we do not attribute the cause (IP / fingerprint / behaviour cannot be distinguished from a response shape alone). Cleared on success.
suspected_rate_limited	Sticky on HTTP 429. Cleared on success.
suspected_ip_burned	Sticky on a soft-block / 403-shaped failure. Factual observation, not a proven cause. Cleared on success.
suspected_fingerprint_rejected	Sticky on a challenge-shaped failure. Factual observation, not a proven cause. Cleared on success.
suspected_shadow_banned	Sticky when responses look superficially OK but fail verification repeatedly. Cleared on success.
last_failure_status	Status code of the most recent failure, or null. Cleared on success.
last_failure_markers	Substring tokens spotted in the most recent failure body or headers (e.g. "datadome", "akamai-sensor", "cf-challenge", "imperva", "access-denied"). Factual evidence only — no inferred cause. Cleared on success.

Data Table

Field	Description
trace_schema_version	Always 4 for this API.
request_id	Unique id for this fetch.
verdict	success, soft_block, hard_block, verifier_failed, transport_error, or budget_exhausted.
usable_content	Whether the verifier judged response.body usable.
response	{ status, headers, body, final_url, screenshot? }. status is 0 for transport-level failures; screenshot is base64 PNG when enabled.
final_signal	Classifier signal: { kind, confidence, waf?, … } (kind ∈ success, challenge, soft_block, hard_block, auth_required, not_found, geo_block, transport_error, unknown).
verification	Verifier verdict — present only when a verifier ran.
decision_trace	One entry per attempt: strategy_id (impit_chrome, chromium_browser, anubis_pow), transport (http/browser/solver), browser, outcome, latency_ms, cost_units. The winning tier is the successful entry.
strategy_learning	Strategy-selection / bandit trace — present when learning is active.
cost	{ attempt_count, cost_units, latency_ms, browser_latency_ms, rotation_count }. cost.latency_ms is end-to-end handler latency.
session	{ original_session, effective_session, rotation_count, diagnostics } — see Session diagnostics. Present when a session is in play.

This standby Actor returns JSON directly from the HTTP endpoint. It does not write per-request results to an Apify dataset by default.

Session-outcome log and optional dataset push

Every call emits one structured session-outcome log line summarising the request, the resolved tier, and the post-call session diagnostics. This is always on and useful for grepping the Apify run log.

Set GHOST_FETCH_SESSION_DATASET to opt into pushing the same record into an Apify dataset:

• unset / 0 / false → no push (default)
• 1 / true → default dataset
• any other value → treated as a dataset name or ID

Dataset push is fire-and-forget: failures are logged with a warning and swallowed so a dataset outage cannot brick the standby Actor.

Local Runtime Caveat

The container image is the deployment gate. It pulls the patched Chromium runtime needed for the browser tier from the KV store at cold start.

The host-local dev runtime can fail independently, e.g. around the patched Chromium runtime not being available locally. If the local browser tier fails but the container smoke passes, the deployable path is considered green.

See docs/operations.md for exact Podman/Docker commands.

Cost Estimation

There is no per-event charge logic in this Actor. Runtime cost comes from normal Apify compute usage and any proxy traffic you enable.

Tier 1 is the cheap path and usually finishes in milliseconds to a few seconds. Tier 2 starts or reuses a browser and can take several seconds or more, especially on first use of a country/session combination.

Cookies and session jar

When you pass session, ghost-fetch threads the session through every layer so cookies, IP, and TLS fingerprint stay consistent across calls:

caller (session="X", country="CA")
   │
   ▼
ghost-fetch server
   ├─► impit pool          keyed by (country, session)
   │     └─ same Impit instance reused across calls
   │     └─ cookieJar adapter on each instance writes/reads `_jars`
   │
   ├─► ghost browser tier   per-session browser context
   │     └─ snapshots context.cookies() into `_jars` on success
   │
   ├─► cookie jar (`_jars`) keyed by `${country}:${session}`
   │     └─ single source of truth, written by both tiers, read by both
   │
   └─► Apify residential proxy
         └─ `session-X` selector → sticky residential exit IP

What this gives you:

• Same session → same exit IP (Apify residential session stickiness).
• Same (country, session) → same cookie jar; cookies minted by either tier are visible to the other on subsequent calls.
• Same (country, session) → same Impit instance with the same TLS impersonation profile — fingerprint-binding WAFs (DataDome, some Akamai BMP rulesets) don't see a JA4 swap between calls.

The cookie jar is bidirectional:

• ghost → jar: context.cookies() snapshot at end of a successful navigation (session-window.ts).
• impit → jar: Set-Cookie from every impit response captured via the cookie jar adapter wired at instance construction (http-fetch.ts + impit-cookie-adapter.ts).
• jar → ghost: Playwright context inherits jar cookies before nav.
• jar → impit: outbound Cookie header built from jar via cookieHeader(key, url).

Without this, impit would drop every Set-Cookie response on the floor (impit's default — no cookie store). Sites that rotate their identifier cookie per response (canonically DataDome's datadome=…) become defeated for Tier 1 amortization: the next call replays a stale value, 4xx's, and the cascade falls back to ghost_nav every time. With the adapter, Tier 1 sees the freshest cookie state on every call.

Identity bundling — one JA4 across both tiers

(country, session) binds the residential exit IP and the cookie jar. The TLS fingerprint (JA4) follows automatically: chromium is the only browser tier, so Tier 1 (the impit chrome142 preset) and Tier 2 (the ghost chromium) present the same chrome-family JA4. Cookies a ghost attempt mints replay cleanly on the Tier 1 path, so warm calls stay on the cheap HTTP-impersonate tier.

There is no per-session engine selection to record or resolve: with a single browser family, the cross-engine JA4 mismatch that a firefox/chrome split could cause on a shared jar cannot happen. render.browser accepts only chrome; a firefox request is rejected by the schema, not silently served by chromium.

The browser family each attempt dispatched against is recorded per-attempt in the response's decision_trace[].browser (always "chrome"). The cookie-replay path is otherwise unobservable from the response.

Origin-aware session identity

The cookie jar is still keyed by (country, session) and relies on domain/path filtering for correctness. Recovery locks and session quality are keyed more narrowly by (country, session, target origin). That lets a shared Crawlee session keep using a healthy origin even if another origin under the same session has failed.

For session-backed calls the response includes a session diagnostics object with the origin key, quality score, consecutive failures, and basic suspicion flags. Crawlee adapters can use this to coordinate per-origin retirement without throwing away the whole session for unrelated hosts.

Known WAF behaviors

• DataDome single-uses some path-scoped cookies. Even with the adapter writing fresh values, datadome= issued for a PDP GET is typically rejected on the next PDP GET in the same session — the WAF treats it as "already consumed." XHR endpoints on the same origin (e.g. JSON price APIs) tend to accept the same rotated cookie, so the adapter is the difference between a working POST and a 403-loop POST. The PDP GET stays on ghost_nav regardless.
• Akamai BMP _abck rotates per request, accepts replay. Tier 1 amortization works cleanly here.
• Cloudflare cf_clearance is long-lived after a single ghost_nav mint; subsequent calls ride Tier 1 for the cookie's full TTL (minutes to an hour).

Documentation

• docs/api-v1-fetch.md
• docs/trace-v4.md
• docs/architecture.md
• docs/operations.md
• docs/sessions-and-storage.md
• docs/strategy-learning.md
• docs/telemetry.md
• docs/profiles.md
• docs/waf-fingerprinting.md
• docs/smoke-targets.md
• docs/roadmap.md

Useful Commands

npm test
npm run build
npm run lint
npm run format:check
podman build -t ghost-fetch:local .

Run smoke against a running server:

node test/smoke-http.mjs https://example.com
npm run smoke:matrix

Extract sanitized passive WAF fingerprint candidates from a saved response or HAR without committing the capture:

$npm run fingerprint:response -- ./capture.har

Tips And Advanced Options

• Use render: { mode: "http" } when the caller can handle blocked pages itself (Tier 1 only, no browser fallback).
• Use render: { mode: "browser" } when you already know the target always needs a browser.
• Set country for geo-sensitive targets instead of relying on random proxy exits.
• Set session when multiple calls should share cookies and proxy stickiness.
• Keep BROWSER_POOL_MAX low on smaller memory allocations.

Useful environment variables:

Variable	Purpose
APIFY_META_ORIGIN	Must be STANDBY locally and on platform.
ACTOR_WEB_SERVER_PORT	HTTP port. Default: 3001.
APIFY_PROXY_PASSWORD	Enables Apify proxy usage when set. Required in gateway mode (the HTTP/moat tier dials the leased IP with it) unless GHOST_GATEWAY_HTTP=disabled; otherwise boot fails fast.
GHOST_GATEWAY_URL	Gateway base URL. Set ⇒ gateway mode (browser tier over CDP); unset ⇒ lean (HTTP-only, no browser tier).
GHOST_GATEWAY_HTTP	disabled ⇒ explicit browser-only gateway deploy (no cheap-HTTP moat tier; the only sanctioned secret-less gateway deploy). Default: enabled.
APIFY_PROXY_HOST	Proxy host. Default: proxy.apify.com.
APIFY_PROXY_PORT	Proxy port. Default: 8000.
APIFY_PROXY_USER	Raw proxy-user override; wins over GHOST_FETCH_PROXY_GROUP. Default: unset (group selector applies).
GHOST_FETCH_PROXY_GROUP	Deploy-pinned proxy group: RESIDENTIAL (default) or DATACENTER/DC → auto. See caveat below.
PROXY_COUNTRY	Fallback country when request country is omitted.
BROWSER_POOL_MAX	Max warm Tier-1 (impit/HTTP) pool entries. Default: 8. (The chromium browser is a singleton.)
CHROMIUM_BROWSER_CEILING_MB	Per-Browser memory ceiling used to size the Chromium pool. Default: 1280.
GHOST_BINARY_KV_STORE_ID	Optional KV store ID for the patched Chromium runtime bundle.
GHOST_BINARY_KV_KEY	Optional KV record key. Default: runtime-tar-zst.
GHOST_FETCH_SESSION_DATASET	Opt into pushing the session-outcome record into an Apify dataset. See Session-outcome log and optional dataset push.
GHOST_FETCH_TELEMETRY	Opt into sanitized trace-v4 telemetry (1 / true / yes). Off by default. See docs/telemetry.md.
GHOST_FETCH_TELEMETRY_DATASET_ID	Dataset for telemetry. Valid Apify id/name routes there; malformed value warns and falls back to the default ghost-fetch-telemetry.
GHOST_FETCH_TELEMETRY_RAW	Opt into attaching the full raw fetch result (incl. HTML body) to telemetry. Requires a valid custom GHOST_FETCH_TELEMETRY_DATASET_ID; never writes raw payloads to the default dataset. See docs/telemetry.md#raw-response-capture.
GHOST_FETCH_TELEMETRY_RAW_MAX_BYTES	Max raw HTML body bytes attached per event. Default: 6000000 (fits Apify's ~9 MB item limit).

Proxy group caveat (GHOST_FETCH_PROXY_GROUP=DATACENTER). Datacenter is cheaper but its country coverage is narrower than residential, and US-state targeting is residential-only. A country (or PROXY_COUNTRY) with no datacenter IPs fails the connection — the group is deploy-pinned, so there is no runtime datacenter→residential escalation; once the recovery ladder is exhausted within the pinned group, ghost-fetch returns a plain block error. Pin datacenter only when recon shows the target tolerates it. APIFY_PROXY_USER, if set, overrides this toggle with its raw token.

The Docker image pulls the patched Chromium runtime from a KV store at cold start (GHOST_BINARY_KV_STORE_ID / GHOST_BINARY_KV_KEY); there is no stock browser baked into the image.

FAQ, Disclaimers, And Support

Does ghost-fetch parse the page? No. It returns the fetched body and leaves parsing to the caller.

Does it solve captchas? No. It handles passive challenge pages best. Manual or interactive captcha solving is out of scope.

Can I use it for any website? Use it only where you have a lawful basis and where your usage respects the target site's terms, robots rules, and rate limits.

Where should issues go? Use the Apify Issues tab or the repository issue tracker with the target URL, input JSON, the verdict, final_signal.kind, and the decision_trace (which strategies ran and their outcomes).

Usage Boundaries

Use ghost-fetch only where you have a lawful basis and where your usage respects the target site's terms, robots rules, and rate limits.