# Batch Anti-Bot Detection API - Profile Many URLs in One Run (`fenlo-services/batch-recon`) Actor Detect the anti-bot stack (Cloudflare, DataDome, Akamai, PerimeterX, Imperva) for many URLs at once and return one JSON object of per-URL defense fingerprints (each with a recommended\_action) plus a separate errors list. - **URL**: https://apify.com/fenlo-services/batch-recon.md - **Developed by:** [Fenlo AI](https://apify.com/fenlo-services) (community) - **Categories:** Automation, Developer tools, Agents - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $5.00 / 1,000 url profileds This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Batch Anti-Bot Detection API — Profile a Fleet of URLs in One Run Programmatic, **batch detection** of the anti-bot stack protecting many websites at once. Hand it a list of URLs and get back one structured dataset row per URL: a defense fingerprint for your own scraper fleet on every reachable URL, and a tagged error row for any URL that could not be reached. This is the many-URL companion to the single-URL **recon-meta** Actor. Same detection, same honest output — fanned out concurrently so you can pre-flight a whole crawl frontier in a single run instead of calling a one-URL endpoint in a loop. This is a **detection-as-intelligence** tool. It reports *what* defenses each site runs so you can plan with your own tooling. It does not provide instructions for getting past those defenses, and it makes no promise of access. ### What it detects (per URL) - The stack: Cloudflare, DataDome, Akamai, PerimeterX, Imperva, or none - A confidence score and the raw signals that triggered the match - A descriptive `defense_profile` (for example, *"Cloudflare present; expect an interactive JS challenge"*) - A coarse `difficulty_estimate`: `easy` | `medium` | `hard` | `unlikely` - `js_challenge_likely`: whether an interactive JS challenge is expected - `recommended_profile`: the request profile the detection engine would select - `recommended_action`: an honest, machine-branchable hint for how to **approach the target with your own tooling** — `standard_request`, `browser_render_recommended`, or `high_defense_expect_friction`. It describes the kind of request to plan for; it never describes getting past a defense and is not a promise of access. ### Input | Field | Type | Default | Description | |-------|------|---------|-------------| | `urls` | array of string | — | The URLs to fingerprint. | | `concurrency` | integer | `5` | Maximum number of URLs probed at the same time. | | `deep_playwright` | boolean | `false` | If the cheap HTTP probe cannot identify a URL's stack, render that page in a real browser and re-inspect. Slower, costs more per URL. | ### Output **One dataset item per URL.** Each URL the run processes writes exactly one row to the default dataset, so the dataset has the same number of items as the input has URLs. Successful rows carry the full detection record (the same shape the `recon-meta` Actor returns) tagged `"ok": true`; a URL that could not be reached writes a `{ "url", "error", "ok": false }` row instead, so one bad URL never sinks the batch. Branch on the `ok` field to tell a success from a failure. Successful rows are written first, then any failed rows. Successful row: ```json { "url": "https://shop.example.com", "stack": "cloudflare", "confidence": 0.95, "recommended_profile": "stealth", "recommended_action": "browser_render_recommended", "signals": ["header:cf-ray"], "defense_profile": "Cloudflare present; expect an interactive JS challenge (e.g. Turnstile) on protected routes.", "js_challenge_likely": true, "difficulty_estimate": "hard", "ok": true } ```` Failed row: ```json { "url": "https://unreachable.example.com", "error": "ConnectError: ...", "ok": false } ``` ### How it works Each URL is probed with a lightweight HTTP request that inspects the response headers and the first few KB of the body for known signatures. Only when that probe cannot identify the stack **and** you set `deep_playwright: true` does that URL escalate to a full browser render and re-inspect — keeping the common path cheap and fast. URLs are processed concurrently under a bounded semaphore (`concurrency`), so a large list finishes quickly without launching everything at once. A URL that fails to respond is written as its own tagged error row (`ok: false`) and the rest of the batch continues. The per-URL dataset is the point: feed your whole URL frontier in, and let your pipeline branch on `recommended_action` and `difficulty_estimate` per row instead of checking sites one at a time by hand. ### Bring your own proxy For an actual fetch (not just detection), pair this with the companion **stealth-scraper** Actor, which accepts a single proxy parameter. ### Pricing Pay-per-event, billed **per URL**: the run writes one dataset item per URL, and each item is one billable event. A deep browser render costs more because it runs a real browser, and only the URLs that need it incur that cost. ### Build & deploy **This Actor is self-contained: the Docker build context is this Actor folder.** Apify confines the build context to the Actor's own directory, so a `COPY` cannot reach outside it. This Actor depends on the repo-local `fenlo_engine` package (source at `packages/engine` — outside this folder, and home to the shared detection logic that both recon Actors reuse). Because Docker cannot `COPY` across folders, the engine is vendored here as a pre-built pip wheel (`fenlo_engine-*.whl`). Build the wheel into this folder first, then build the image with this folder as the context: ```bash ## From the repo root: build the engine wheel into a throwaway dist/ and copy it ## into this Actor dir. Do NOT use `-o actors/batch-recon`: `uv build` writes a ## `.gitignore` containing `*` into its output dir, which would re-ignore this ## Actor's .actorignore and break `apify push`. dist/ is already git-ignored. rm -f actors/batch-recon/fenlo_engine-*.whl uv build --wheel packages/engine -o dist cp dist/fenlo_engine-*.whl actors/batch-recon/ ## From inside this Actor dir (context = ".") cd actors/batch-recon docker build -t fenlo-batch-recon . ``` The `Dockerfile` (referenced from `.actor/actor.json`) then: 1. installs the vendored engine with `pip install fenlo_engine-*.whl`, and 2. installs this Actor's `requirements.txt` (the Apify SDK), so `import fenlo_engine` resolves inside the container. The base image is `apify/actor-python-playwright`, which ships a Chromium matched to the engine's Playwright dependency (used by the optional `deep_playwright` path). **Trade-off:** the wheel is a vendored copy, so **rebuild it whenever `packages/engine` changes** (otherwise the Actor ships a stale engine). The wheel is git-ignored; a `.actorignore` re-includes it so `apify push` still uploads it. To publish on Apify: `cd actors/batch-recon && apify push` (rebuild the wheel first). See `docs/launch/PUBLISH.md`. # Actor input Schema ## `urls` (type: `array`): The list of URLs to fingerprint for anti-bot defenses. One detection result is returned per URL. ## `concurrency` (type: `integer`): Maximum number of URLs to probe at the same time. Higher is faster but uses more resources. ## `deep_playwright` (type: `boolean`): If the lightweight HTTP probe cannot identify a URL's stack, render that page in a real browser and re-inspect. Slower and costs more per URL. ## Actor input object example ```json { "urls": [ "https://www.g2.com", "https://www.zillow.com", "https://www.gnu.org" ], "concurrency": 5, "deep_playwright": false } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "https://www.g2.com", "https://www.zillow.com", "https://www.gnu.org" ] }; // Run the Actor and wait for it to finish const run = await client.actor("fenlo-services/batch-recon").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": [ "https://www.g2.com", "https://www.zillow.com", "https://www.gnu.org", ] } # Run the Actor and wait for it to finish run = client.actor("fenlo-services/batch-recon").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "https://www.g2.com", "https://www.zillow.com", "https://www.gnu.org" ] }' | apify call fenlo-services/batch-recon --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=fenlo-services/batch-recon", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Batch Anti-Bot Detection API - Profile Many URLs in One Run", "description": "Detect the anti-bot stack (Cloudflare, DataDome, Akamai, PerimeterX, Imperva) for many URLs at once and return one JSON object of per-URL defense fingerprints (each with a recommended_action) plus a separate errors list.", "version": "0.1", "x-build-id": "b4lTQurajxiz4PEz9" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/fenlo-services~batch-recon/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-fenlo-services-batch-recon", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/fenlo-services~batch-recon/runs": { "post": { "operationId": "runs-sync-fenlo-services-batch-recon", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/fenlo-services~batch-recon/run-sync": { "post": { "operationId": "run-sync-fenlo-services-batch-recon", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "urls" ], "properties": { "urls": { "title": "Target URLs", "type": "array", "description": "The list of URLs to fingerprint for anti-bot defenses. One detection result is returned per URL.", "items": { "type": "string" } }, "concurrency": { "title": "Concurrency", "minimum": 1, "maximum": 50, "type": "integer", "description": "Maximum number of URLs to probe at the same time. Higher is faster but uses more resources.", "default": 5 }, "deep_playwright": { "title": "Deep browser check", "type": "boolean", "description": "If the lightweight HTTP probe cannot identify a URL's stack, render that page in a real browser and re-inspect. Slower and costs more per URL.", "default": false } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```