Batch Anti-Bot Detection API — Profile a Fleet of URLs in One Run

Programmatic, batch detection of the anti-bot stack protecting many websites at once. Hand it a list of URLs and get back one structured dataset row per URL: a defense fingerprint for your own scraper fleet on every reachable URL, and a tagged error row for any URL that could not be reached.

This is the many-URL companion to the single-URL recon-meta Actor. Same detection, same honest output — fanned out concurrently so you can pre-flight a whole crawl frontier in a single run instead of calling a one-URL endpoint in a loop.

This is a detection-as-intelligence tool. It reports what defenses each site runs so you can plan with your own tooling. It does not provide instructions for getting past those defenses, and it makes no promise of access.

What it detects (per URL)

• The stack: Cloudflare, DataDome, Akamai, PerimeterX, Imperva, or none
• A confidence score and the raw signals that triggered the match
• A descriptive defense_profile (for example, "Cloudflare present; expect an interactive JS challenge")
• A coarse difficulty_estimate: easy | medium | hard | unlikely
• js_challenge_likely: whether an interactive JS challenge is expected
• recommended_profile: the request profile the detection engine would select
• recommended_action: an honest, machine-branchable hint for how to approach the target with your own tooling — standard_request, browser_render_recommended, or high_defense_expect_friction. It describes the kind of request to plan for; it never describes getting past a defense and is not a promise of access.

Input

Field	Type	Default	Description
urls	array of string	—	The URLs to fingerprint.
concurrency	integer	5	Maximum number of URLs probed at the same time.
deep_playwright	boolean	false	If the cheap HTTP probe cannot identify a URL's stack, render that page in a real browser and re-inspect. Slower, costs more per URL.

Output

One dataset item per URL. Each URL the run processes writes exactly one row to the default dataset, so the dataset has the same number of items as the input has URLs. Successful rows carry the full detection record (the same shape the recon-meta Actor returns) tagged "ok": true; a URL that could not be reached writes a { "url", "error", "ok": false } row instead, so one bad URL never sinks the batch. Branch on the ok field to tell a success from a failure.

Successful rows are written first, then any failed rows.

Successful row:

{
  "url": "https://shop.example.com",
  "stack": "cloudflare",
  "confidence": 0.95,
  "recommended_profile": "stealth",
  "recommended_action": "browser_render_recommended",
  "signals": ["header:cf-ray"],
  "defense_profile": "Cloudflare present; expect an interactive JS challenge (e.g. Turnstile) on protected routes.",
  "js_challenge_likely": true,
  "difficulty_estimate": "hard",
  "ok": true
}

Failed row:

{
  "url": "https://unreachable.example.com",
  "error": "ConnectError: ...",
  "ok": false
}

How it works

Each URL is probed with a lightweight HTTP request that inspects the response headers and the first few KB of the body for known signatures. Only when that probe cannot identify the stack and you set deep_playwright: true does that URL escalate to a full browser render and re-inspect — keeping the common path cheap and fast.

URLs are processed concurrently under a bounded semaphore (concurrency), so a large list finishes quickly without launching everything at once. A URL that fails to respond is written as its own tagged error row (ok: false) and the rest of the batch continues.

The per-URL dataset is the point: feed your whole URL frontier in, and let your pipeline branch on recommended_action and difficulty_estimate per row instead of checking sites one at a time by hand.

Bring your own proxy

For an actual fetch (not just detection), pair this with the companion stealth-scraper Actor, which accepts a single proxy parameter.

Pricing

Pay-per-event, billed per URL: the run writes one dataset item per URL, and each item is one billable event. A deep browser render costs more because it runs a real browser, and only the URLs that need it incur that cost.

Build & deploy

This Actor is self-contained: the Docker build context is this Actor folder. Apify confines the build context to the Actor's own directory, so a COPY cannot reach outside it.

This Actor depends on the repo-local fenlo_engine package (source at packages/engine — outside this folder, and home to the shared detection logic that both recon Actors reuse). Because Docker cannot COPY across folders, the engine is vendored here as a pre-built pip wheel (fenlo_engine-*.whl). Build the wheel into this folder first, then build the image with this folder as the context:

# From the repo root: build the engine wheel into a throwaway dist/ and copy it
# into this Actor dir. Do NOT use `-o actors/batch-recon`: `uv build` writes a
# `.gitignore` containing `*` into its output dir, which would re-ignore this
# Actor's .actorignore and break `apify push`. dist/ is already git-ignored.
rm -f actors/batch-recon/fenlo_engine-*.whl
uv build --wheel packages/engine -o dist
cp dist/fenlo_engine-*.whl actors/batch-recon/

# From inside this Actor dir (context = ".")
cd actors/batch-recon
docker build -t fenlo-batch-recon .

The Dockerfile (referenced from .actor/actor.json) then:

1. installs the vendored engine with pip install fenlo_engine-*.whl, and
2. installs this Actor's requirements.txt (the Apify SDK),

so import fenlo_engine resolves inside the container. The base image is apify/actor-python-playwright, which ships a Chromium matched to the engine's Playwright dependency (used by the optional deep_playwright path).

Trade-off: the wheel is a vendored copy, so rebuild it whenever packages/engine changes (otherwise the Actor ships a stale engine). The wheel is git-ignored; a .actorignore re-includes it so apify push still uploads it.

To publish on Apify: cd actors/batch-recon && apify push (rebuild the wheel first). See docs/launch/PUBLISH.md.