Fetch Strategy Router

A lightweight machine-to-machine router that helps AI agents, crawlers, and automation pipelines choose the cheapest viable way to fetch a URL before spending money on a browser or proxy.

Give it one or many public URLs. For each URL, it performs a bounded direct HTTP probe and returns a routing recommendation.

Routing outcomes

• direct_http — static HTTP is likely sufficient.
• browser — the response looks like a JavaScript shell or nearly empty app page.
• proxy_likely_needed — the site returned a block such as HTTP 403 without a strong browser-challenge signal.
• browser_plus_proxy — challenge / anti-bot signals suggest both browser rendering and alternate network egress may be required.
• retry_later — rate limit, transient server error, or timeout.
• skip — invalid URL, private target, missing resource, authentication requirement, or another condition where escalation is unlikely to help.

Why use it

Browser scraping can be orders of magnitude heavier than a simple HTTP request. This Actor is designed to run first and help downstream agents decide whether to:

1. use Cheerio / HTTP,
2. escalate to Playwright or Puppeteer,
3. add proxy infrastructure,
4. retry later,
5. skip the target.

Typical uses include:

• web-crawling orchestration
• AI-agent tool routing
• scraper cost optimization
• bulk URL triage
• anti-bot preflight checks
• ETL and enrichment pipelines
• crawler fallback logic

What it inspects

• HTTP status
• redirect chain
• response time
• response content type
• bounded response body
• visible static text density
• JavaScript shell indicators
• empty application-root indicators
• Cloudflare signals
• Akamai signals
• DataDome signals
• PerimeterX signals
• Imperva signals
• Sucuri signals
• generic bot/challenge text
• rate-limit headers

Example input

{
  "urls": [
    "https://apify.com",
    "https://www.cloudflare.com"
  ],
  "requestTimeoutSecs": 12,
  "maxConcurrency": 25,
  "maxHtmlBytes": 500000
}

Example output

{
  "requestedUrl": "https://example.com/",
  "finalUrl": "https://example.com/",
  "domain": "example.com",
  "statusCode": 200,
  "responseTimeMs": 184,
  "recommendedRoute": "direct_http",
  "confidence": 0.94,
  "reasons": [
    "substantial_static_text",
    "healthy_static_text_density"
  ],
  "antiBotDetected": false,
  "antiBotVendor": null,
  "antiBotSignals": [],
  "staticSignals": {
    "textChars": 7842,
    "textDensity": 0.118,
    "scriptCount": 9,
    "scriptRatio": 0.12,
    "hasAppRootSignal": false,
    "hasNoscriptJsMessage": false,
    "metaRefresh": false,
    "likelyJsShell": false,
    "htmlLooksValid": true
  },
  "error": null
}

Cost-conscious design

This Actor deliberately does not launch a browser, solve CAPTCHAs, call an LLM, or use residential proxies. It reads a bounded amount of public response data and makes one main request per URL, following only a small redirect chain.

That makes it suitable as a first-stage router before more expensive Actors.

Security

Only public HTTP and HTTPS targets are accepted. Hostnames are resolved before requests and again before redirects. Loopback, private, link-local, and reserved IP ranges are rejected to reduce SSRF risk.

Interpretation

A routing recommendation is a heuristic, not a guarantee. Websites can vary responses by geography, IP reputation, cookies, TLS fingerprint, user agent, and time. The Actor is intended to choose a sensible next step cheaply, not to prove that a specific scraping technique will always work.

Output

One dataset item is produced for every submitted URL, including blocked or failed URLs. This makes the output convenient for APIs, MCP tools, n8n, Make, Zapier, custom crawlers, and autonomous agents.