URL Retrieval Preflight

What does URL Retrieval Preflight do?

URL Retrieval Preflight recommends the lowest-cost likely way to retrieve a public HTTPS URL and returns the deterministic evidence that produced the decision. It is a small decision primitive for agents and pipelines that must choose between plain HTTP, a browser render, a PDF/feed/JSON/XML parser, or walking away from an authenticated, blocked, or unsupported target. It does not use an LLM and never returns an opaque score.

The Actor starts with a bounded HEAD probe and, only when the headers cannot settle the decision (method not allowed, missing or ambiguous content type, HTML that might depend on JavaScript, or generic XML that might be a feed), performs a GET bounded to the first 64 KiB after decompression. Redirects and transient retries stay within strict request, time, and byte budgets. You can invoke it from the Apify API, scheduled runs, integrations, or Apify MCP and monitor every run through the Apify Console.

Why use URL Retrieval Preflight?

• Decide how to fetch a URL before spending money on the wrong retrieval tool.
• Replace ad-hoc HEAD/GET probing, redirect handling, content-type sniffing, and JS-shell detection with one stable JSON contract.
• Receive the recommendation plus structured evidence: status, content type, redirect history, and the specific markers that fired.
• Keep routing deterministic and reproducible without runtime AI or a paid data supplier.
• Block localhost, private networks, link-local addresses, metadata services, unsafe redirects, oversized responses, and slow requests by default.

Example agent queries include “How should I fetch this URL?”, “Does this page need a browser or plain HTTP?”, “Is this URL a PDF, a feed, or a JSON API?”, and “Check whether this link requires authentication before I crawl it.”

How to preflight a URL

1. Open the Actor's Input tab.
2. Enter one public HTTPS URL.
3. Start the Actor and inspect the single dataset result.

Always ensure your retrieval and use of the target complies with its terms, robots directives, copyright rules, and applicable law.

Input

See the Input tab for the full schema. The contract deliberately has a single field:

{
  "url": "https://example.com/article"
}

Output

The Actor writes exactly one versioned JSON envelope. You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

{
  "contractVersion": "1.0",
  "ok": true,
  "request": { "url": "https://example.com/article" },
  "probe": {
    "method": "head+get",
    "requestedUrl": "https://example.com/article",
    "finalUrl": "https://example.com/article",
    "httpStatus": 200,
    "contentType": "text/html; charset=utf-8",
    "sniffBytes": 65536,
    "truncated": true,
    "contentHash": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
    "retrievedAt": "2026-08-11T06:00:00.000Z",
    "durationMs": 184,
    "redirectChain": []
  },
  "recommendation": {
    "strategy": "plain_http",
    "evidence": [
      {
        "type": "http_response",
        "source": "https://example.com/article",
        "retrievedAt": "2026-08-11T06:00:00.000Z",
        "tier": "direct",
        "response": {
          "status": 200,
          "contentType": "text/html; charset=utf-8",
          "finalUrl": "https://example.com/article"
        }
      },
      {
        "type": "http_header",
        "source": "https://example.com/article",
        "retrievedAt": "2026-08-11T06:00:00.000Z",
        "tier": "direct",
        "field": "content-type",
        "value": "text/html; charset=utf-8"
      },
      {
        "type": "content_sniff",
        "source": "https://example.com/article",
        "retrievedAt": "2026-08-11T06:00:00.000Z",
        "tier": "direct",
        "field": "readable_text_characters",
        "value": 4312
      }
    ]
  },
  "warnings": []
}

Strategy	Triggering evidence (summary)
plain_http	Readable text is present in the first 64 KiB of the HTML/text response
browser_render	HTML with little readable text plus JS-dependence markers (listed in evidence)
pdf_parser	application/pdf content type or %PDF- body signature
feed_parser	RSS/Atom/JSON-Feed content type or an RSS/Atom root element
json_api	JSON content type (including +json suffixes) or a valid JSON body
xml_parser	Generic XML inspected without RSS/Atom feed markers
authentication_required	HTTP 401, or 403 with a www-authenticate challenge
blocked	404/410, bare 403, 429, or server responses — the URL cannot be retrieved as-is
unsupported	Binary or otherwise unhandled media types (archives, images, audio, video, fonts)

Field	Type	Meaning
ok	boolean	Whether a preflight determination completed
probe	object	Method used, final URL, status, content type, byte count, timestamp, redirects
recommendation	object	Strategy plus the evidence entries that produced it
error	object	Stable machine-readable free failure when ok is false

How much does URL Retrieval Preflight cost?

The configured price is one $0.01 url-preflight pay-per-event charge for a completed determination — including blocked and unsupported outcomes, which are valid answers. Network, timeout, security-policy, resource-budget, invalid-input, and internal failures are not charged. A fixed 20-run private Apify sample measured a p95 platform cost of $0.0001029against the$0.004 release ceiling.

Accuracy and advanced options

The recommendation is only as current as the probe; sites can vary responses by client, geography, or time. browser_render is a routing hint derived from explicit structural markers, not a guarantee that rendering will succeed. Query values are used for retrieval but redacted from output and logs.

FAQ, responsible use, and support

Does plain_http guarantee the full page is readable?

No. It means the probed prefix contains substantive readable text. A page can still lazy-load sections or paginate content that the 64 KiB sniff does not cover.

Why is 404 reported as blocked instead of an error?

Because “this URL is not retrievable” is a valid preflight answer. The envelope stays ok: true, the strategy is blocked, and the HTTP status is included as evidence. Errors are reserved for failures of the Actor itself (DNS, timeouts, redirect-policy failures, and resource limits). If a required GET probe fails, the Actor emits a free structured failure instead of guessing from insufficient HEAD evidence.

The Actor reads public response headers and content prefixes and does not intentionally collect private user data. Use results only with a lawful purpose and consult qualified counsel when unsure. Use the Actor's Issues tab for bug reports and the API tab for programmatic invocation.