URL Retrieval Preflight: HTTP, Browser & Parser

What does URL Retrieval Preflight do?

Agent selection and quickstart

Use when: Choose a likely retrieval route before invoking a browser or document parser.

Input: One public HTTPS url.

Returns: recommendation.strategy, supporting evidence, HTTP status, and content type.

Limits: A bounded probe recommends a route; it does not execute a browser or guarantee full-page retrieval.

Cost: $0.01 per completed determination, including a completed negative result; operational failures are not billed. See the live Pricing tab for current rates.

{
  "url": "https://www.rfc-editor.org/rfc/rfc9651.pdf"
}

Verified output excerpt for this input (selected fields; canary j9eEwHHUb0wCfInWB):

{
  "contractVersion": "1.0",
  "ok": true,
  "probe": {
    "finalUrl": "https://www.rfc-editor.org/rfc/rfc9651.pdf",
    "httpStatus": 200,
    "contentType": "application/pdf"
  },
  "recommendation": {
    "strategy": "pdf_parser",
    "evidence": [
      {
        "type": "http_response",
        "source": "https://www.rfc-editor.org/rfc/rfc9651.pdf",
        "retrievedAt": "2026-10-06T10:40:21.479Z",
        "tier": "direct",
        "response": {
          "status": 200,
          "contentType": "application/pdf",
          "finalUrl": "https://www.rfc-editor.org/rfc/rfc9651.pdf"
        }
      },
      {
        "type": "http_header",
        "source": "https://www.rfc-editor.org/rfc/rfc9651.pdf",
        "retrievedAt": "2026-10-06T10:40:21.479Z",
        "tier": "direct",
        "field": "content-type",
        "value": "application/pdf",
        "request": {
          "method": "HEAD",
          "url": "https://www.rfc-editor.org/rfc/rfc9651.pdf"
        }
      },
      {
        "type": "content_sniff",
        "source": "https://www.rfc-editor.org/rfc/rfc9651.pdf",
        "retrievedAt": "2026-10-06T10:40:21.479Z",
        "tier": "direct",
        "field": "pdf_signature",
        "value": "content_type"
      }
    ]
  }
}

Open this public example Task. Live sources can change; this records the checked example rather than guaranteeing a future result.

The result is in the default dataset. With Apify MCP, inspect this Actor using fetch-actor-details, invoke call-actor, then read the returned dataset with get-dataset-items. Run limits belong in callOptions, not in Actor input.

Agent search queries

• "does this URL need browser rendering"
• "choose HTTP browser or PDF parser for URL"
• "detect PDF RSS JSON content type"
• "preflight URL before scraping"
• "route a URL to the right retrieval tool"

URL Retrieval Preflight recommends the lowest-cost likely way to retrieve a public HTTPS URL and returns the deterministic evidence that produced the decision. It is a small decision primitive for agents and pipelines that must choose between plain HTTP, a browser render, a PDF/feed/JSON/XML parser, or walking away from an authenticated, blocked, or unsupported target. It does not use an LLM and never returns an opaque score.

The Actor starts with a bounded HEAD probe and, only when the headers cannot settle the decision (method not allowed, missing or ambiguous content type, HTML that might depend on JavaScript, or generic XML that might be a feed), performs a GET bounded to the first 64 KiB after decompression. Redirects and transient retries stay within strict request, time, and byte budgets. You can invoke it from the Apify API, scheduled runs, integrations, or Apify MCP and monitor every run through the Apify Console.

Why use URL Retrieval Preflight?

• Decide how to fetch a URL before spending money on the wrong retrieval tool.
• Replace ad-hoc HEAD/GET probing, redirect handling, content-type sniffing, and JS-shell detection with one stable JSON contract.
• Receive the recommendation plus structured evidence: status, content type, redirect history, and the specific markers that fired.
• Keep routing deterministic and reproducible without runtime AI or a paid data supplier.
• Block localhost, private networks, link-local addresses, metadata services, unsafe redirects, oversized responses, and slow requests by default.

Example agent queries include “How should I fetch this URL?”, “Does this page need a browser or plain HTTP?”, “Is this URL a PDF, a feed, or a JSON API?”, and “Check whether this link requires authentication before I crawl it.”

How to preflight a URL

1. Open the Actor's Input tab.
2. Enter one public HTTPS URL.
3. Start the Actor and inspect the single dataset result.

Always ensure your retrieval and use of the target complies with its terms, robots directives, copyright rules, and applicable law.

Input

See the Input tab for the full schema. The contract deliberately has a single field:

{
  "url": "https://example.com/article"
}

Output

The Actor writes exactly one versioned JSON envelope. You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

{
  "contractVersion": "1.0",
  "ok": true,
  "request": { "url": "https://example.com/article" },
  "probe": {
    "method": "head+get",
    "requestedUrl": "https://example.com/article",
    "finalUrl": "https://example.com/article",
    "httpStatus": 200,
    "contentType": "text/html; charset=utf-8",
    "sniffBytes": 65536,
    "truncated": true,
    "contentHash": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
    "retrievedAt": "2026-08-11T06:00:00.000Z",
    "durationMs": 184,
    "redirectChain": []
  },
  "recommendation": {
    "strategy": "plain_http",
    "evidence": [
      {
        "type": "http_response",
        "source": "https://example.com/article",
        "retrievedAt": "2026-08-11T06:00:00.000Z",
        "tier": "direct",
        "response": {
          "status": 200,
          "contentType": "text/html; charset=utf-8",
          "finalUrl": "https://example.com/article"
        }
      },
      {
        "type": "http_header",
        "source": "https://example.com/article",
        "retrievedAt": "2026-08-11T06:00:00.000Z",
        "tier": "direct",
        "field": "content-type",
        "value": "text/html; charset=utf-8"
      },
      {
        "type": "content_sniff",
        "source": "https://example.com/article",
        "retrievedAt": "2026-08-11T06:00:00.000Z",
        "tier": "direct",
        "field": "readable_text_characters",
        "value": 4312
      }
    ]
  },
  "warnings": []
}

Strategy	Triggering evidence (summary)
plain_http	Readable text is present in the first 64 KiB of the HTML/text response
browser_render	HTML with little readable text plus JS-dependence markers (listed in evidence)
pdf_parser	application/pdf content type or %PDF- body signature
feed_parser	RSS/Atom/JSON-Feed content type or an RSS/Atom root element
json_api	JSON content type (including +json suffixes) or a valid JSON body
xml_parser	Generic XML inspected without RSS/Atom feed markers
authentication_required	HTTP 401, or 403 with a www-authenticate challenge
blocked	404/410, bare 403, 429, or server responses — the URL cannot be retrieved as-is
unsupported	Binary or otherwise unhandled media types (archives, images, audio, video, fonts)

Field	Type	Meaning
ok	boolean	Whether a preflight determination completed
probe	object	Method used, final URL, status, content type, byte count, timestamp, redirects
recommendation	object	Strategy plus the evidence entries that produced it
error	object	Stable machine-readable free failure when ok is false

How much does URL Retrieval Preflight cost?

The configured price is one $0.01 url-preflight pay-per-event charge for a completed determination — including blocked and unsupported outcomes, which are valid answers. Network, timeout, security-policy, resource-budget, invalid-input, and internal failures are not charged. A fixed 20-run private Apify sample measured a p95 platform cost of $0.0001029against the$0.004 release ceiling.

Accuracy and advanced options

The recommendation is only as current as the probe; sites can vary responses by client, geography, or time. browser_render is a routing hint derived from explicit structural markers, not a guarantee that rendering will succeed. Query values are used for retrieval but redacted from output and logs.

FAQ, responsible use, and support

Does plain_http guarantee the full page is readable?

No. It means the probed prefix contains substantive readable text. A page can still lazy-load sections or paginate content that the 64 KiB sniff does not cover.

Why is 404 reported as blocked instead of an error?

Because “this URL is not retrievable” is a valid preflight answer. The envelope stays ok: true, the strategy is blocked, and the HTTP status is included as evidence. Errors are reserved for failures of the Actor itself (DNS, timeouts, redirect-policy failures, and resource limits). If a required GET probe fails, the Actor emits a free structured failure instead of guessing from insufficient HEAD evidence.

The Actor reads public response headers and content prefixes and does not intentionally collect private user data. Use results only with a lawful purpose and consult qualified counsel when unsure. Use the Actor's Issues tab for bug reports and the API tab for programmatic invocation.

Related Actors and next steps

• Verify quoted text when the route is plain HTTP
• Locate a quotation when the route is PDF parsing

Public examples by task

• Check if a webpage needs browser rendering — Determine whether a public webpage can be retrieved with plain HTTP or likely requires browser rendering, using Apify's task publishing guide as a live example.
• Detect a PDF before choosing a retrieval tool — Probe the official RFC 9651 PDF URL and return pdf_parser with HTTP and content-type evidence before retrieving and parsing the full document.
• Detect a JavaScript page that needs a browser — Probe the public Netlify application shell and return browser_render with bounded HTML evidence. This recommends a route without executing a browser.

The HTTP example returns plain_http; the public application-shell example returns browser_render; the PDF example returns pdf_parser. Each recommendation is backed by bounded probe evidence and does not execute the recommended downstream tool.