# Anti-Bot & Proxy Tier Checker (`hebaabdalaziz/anti-bot-proxy-checker`) Actor

Check any URL through direct, datacenter, residential and Google SERP routes. See which work, which anti-bot vendor blocks them (Cloudflare, DataDome, Akamai, AWS WAF, PerimeterX, Kasada), page size, speed, and the cheapest working proxy tier.

- **URL**: https://apify.com/hebaabdalaziz/anti-bot-proxy-checker.md
- **Developed by:** [Heba Abdalaziz](https://apify.com/hebaabdalaziz) (community)
- **Categories:** Developer tools, Automation, SEO tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 result rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does it do?

**Anti-Bot & Proxy Tier Checker** sends one real browser-like request to each URL through each access route you pick, then tells you which routes get the page:

- **Direct:** the Actor's own server IP, no proxy.
- **Datacenter:** Apify datacenter proxy.
- **Residential:** Apify residential proxy, in a country you choose.
- **Google SERP:** Apify Google SERP proxy, for Google Search URLs only (`www.google.<tld>/search?...`).

For every URL and route you get the HTTP status and whether the page really worked or was a challenge page. You also get **which anti-bot vendor** is in front of the site (Cloudflare, DataDome, Akamai, PerimeterX, AWS WAF, Incapsula, Kasada, Google's own wall), plus the response size and speed. A summary row per URL names the **cheapest route that works** and estimates its proxy cost per 1,000 pages.

### Why use it?

- **Scraper developers:** pick the right proxy tier before writing a scraper. Don't pay for residential traffic when datacenter IPs work, and don't fight a DataDome wall with datacenter IPs.
- **Budgeting:** get a rough proxy cost per 1,000 pages from the real page size.
- **SEO and site owners:** check what bot protection your site (or a competitor's) shows to cloud and residential IPs, and whether real pages or challenge pages are served.
- **Monitoring:** schedule it to notice when a target adds or changes bot protection.

### How to use it

1. Paste one or more full URLs, starting with `http://` or `https://`, into **URLs to check** (up to 100 per run).
2. Pick the **Routes to test**. Direct and datacenter are cheapest; add residential or Google SERP when you need them.
3. Optionally set the residential country, browser fingerprint, text markers, attempts per route and timeout.
4. Click **Start**. Most runs finish in seconds.
5. Open the **Summary per URL** view for the answer, or **Results per route** for the details. Download as JSON, CSV or Excel, or read the dataset through the API.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `startUrls` | array of URLs | (required) | Pages to test, max 100 per run. Extra URLs are listed in `FAILED_INPUTS` as skipped. |
| `routes` | array | `["direct", "datacenter"]` | Any of `direct`, `datacenter`, `residential`, `googleSerp`. |
| `residentialCountry` | string | `US` | Two-letter country code for the residential route. |
| `browserFingerprint` | string | `chrome` | `chrome`, `safari`, `firefox` or `edge`: the TLS/HTTP2 fingerprint and default headers each request copies. |
| `markers` | array of strings | `[]` | Case-sensitive strings to look for in each body. The results go in `markersFound`. |
| `attemptsPerRoute` | integer 1-3 | `1` | Retries for a blocked or timed-out route. Each proxy attempt uses a new proxy IP; direct attempts always come from the Actor's server IP. |
| `timeoutSecs` | integer 5-30 | `20` | Max time per request, for every route. Google SERP answers can take 10-20 s, so keep 20 or more for that route. |

```json
{
    "startUrls": [
        { "url": "https://www.bayt.com/en/uae/jobs/" },
        { "url": "https://www.google.com/search?q=web+scraping&hl=en&gl=us" }
    ],
    "routes": ["direct", "datacenter", "residential", "googleSerp"],
    "residentialCountry": "US",
    "browserFingerprint": "chrome",
    "markers": ["</html>"],
    "attemptsPerRoute": 1,
    "timeoutSecs": 20
}
```

The Google SERP route only applies to Google Search URLs: `google.<tld>` or `www.google.<tld>` with a `/search` path. For other URLs, including `news.google.com`, it is listed under `notApplicableRoutes` in the summary row. It is not tested and not charged. The same applies when the SERP proxy itself rejects a URL with HTTP 400.

Private, loopback, link-local and cloud-metadata addresses (for example `10.x`, `192.168.x`, `127.0.0.1`, `169.254.169.254`, `localhost`) are rejected.

### Output

The dataset has one row per URL and route, plus one `"route": "summary"` row per URL. Below are real rows from a cloud run on 2026-10-12 for a Cloudflare-protected job board.

```json
[
  {
    "url": "https://www.bayt.com/en/uae/jobs/",
    "route": "datacenter",
    "status": 403,
    "ok": false,
    "blocked": true,
    "blockReason": "403 Cloudflare challenge",
    "antiBotVendors": ["cloudflare"],
    "responseBytes": 5972,
    "bodyTruncated": false,
    "responseSecs": 0.64,
    "finalUrl": "https://www.bayt.com/en/uae/jobs/",
    "pageTitle": "Just a moment...",
    "hasNextData": false,
    "hasJsonLd": false,
    "proxyCountry": null,
    "fingerprint": "chrome",
    "attempts": 2,
    "checkedAt": "2026-10-12T13:13:58+00:00"
  },
  {
    "url": "https://www.bayt.com/en/uae/jobs/",
    "route": "residential",
    "status": 200,
    "ok": true,
    "blocked": false,
    "blockReason": null,
    "antiBotVendors": ["cloudflare"],
    "responseBytes": 247781,
    "bodyTruncated": false,
    "responseSecs": 1.03,
    "finalUrl": "https://www.bayt.com/en/uae/jobs/",
    "pageTitle": "Jobs in UAE (Oct 2026) - Bayt.com",
    "hasNextData": false,
    "hasJsonLd": true,
    "proxyCountry": "US",
    "fingerprint": "chrome",
    "attempts": 1,
    "checkedAt": "2026-10-12T13:13:58+00:00"
  },
  {
    "url": "https://www.bayt.com/en/uae/jobs/",
    "route": "summary",
    "cheapestWorkingRoute": "residential",
    "workingRoutes": ["residential"],
    "blockedRoutes": ["direct", "datacenter"],
    "notApplicableRoutes": [],
    "antiBotVendors": ["cloudflare"],
    "estimatedCostPer1000Pages": 1.9822,
    "costBasis": "Estimate: response size x $8 per GB (Apify residential proxy list price) x 1,000 pages.",
    "verdict": "Works on residential; cheapest working route: residential. Blocked on direct, datacenter.",
    "checkedAt": "2026-10-12T13:13:58+00:00"
  }
]
```

Other answers from the same test runs:

| Site | Direct | Datacenter | Residential | Detected |
|---|---|---|---|---|
| eBay home | 403 Forbidden (behind Akamai) | 403 Forbidden (behind Akamai) | 200 ok | akamai |
| G2 | 403 DataDome challenge | 403 DataDome challenge | 403 DataDome challenge | datadome, cloudflare |
| Trustpilot review page | 403 AWS WAF challenge | 403 AWS WAF challenge | 403 AWS WAF challenge | aws-waf |
| Hyatt | 429 Kasada challenge | 429 Kasada challenge | 429 Kasada challenge | kasada |
| Zillow home | 200 ok | 200 ok | 200 ok | perimeterx |
| Google search | 200 Google JavaScript wall | (not run) | (not run) | google; the Google SERP route returned results |

URLs that are invalid, do not resolve, or were not checked are listed with a reason in the `FAILED_INPUTS` record of the run's key-value store. Nothing is charged for them.

### Data fields

Fields on every **route row** (`route` = `direct`, `datacenter`, `residential` or `googleSerp`):

| Field | Type | Description |
|---|---|---|
| `url` | string | Input URL, normalized. |
| `route` | string | Route tested. |
| `status` | integer or null | HTTP status of the reported attempt. Null when no response arrived. Codes 590-599 come from the Apify proxy: it could not reach the site. |
| `ok` | boolean | 2xx and not a challenge or block page. |
| `blocked` | boolean | Refused: challenge or block page, block status (403, 429, 451, 503), timeout or dropped connection. A 401, 404, 405 or 500 is a settled answer, not a block, and is not retried. |
| `blockReason` | string or null | Why the page was not usable, e.g. `403 Cloudflare challenge`, `429 Kasada challenge`, `200 Google JavaScript wall`, `timeout`, `body timeout (headers received, body stalled)`, `404 Not Found`. A vendor is named as the blocker only when its challenge or block page (or a `cf-mitigated` header) was seen; otherwise you get e.g. `403 Forbidden (behind Cloudflare)`. Null when `ok`. |
| `antiBotVendors` | array | Vendors fingerprinted in headers, cookie names or body: `datadome`, `perimeterx`, `kasada`, `aws-waf`, `incapsula`, `akamai`, `cloudflare`, `google`. A vendor can be present without blocking. |
| `responseBytes` | integer or null | Decompressed body size, capped at 3 MB. |
| `bodyTruncated` | boolean | True when the download was stopped early: the body passed 3 MB, or the timeout hit after the headers had arrived. |
| `responseSecs` | number or null | Seconds until the download finished, or until the error. |
| `finalUrl` | string or null | URL after redirects. |
| `pageTitle` | string or null | The page `<title>`. |
| `hasNextData` | boolean or null | Page contains Next.js `__NEXT_DATA__` JSON. |
| `hasJsonLd` | boolean or null | Page contains `application/ld+json` structured data. |
| `markersFound` | object or null | Only when `markers` were given: `{marker: true/false}`. |
| `proxyCountry` | string or null | Requested residential country (residential rows only). |
| `fingerprint` | string | Browser fingerprint used. |
| `attempts` | integer | Requests made on this route. Proxy attempts each used a new proxy IP. |
| `checkedAt` | string | ISO 8601 time in UTC. |

Fields on every **summary row** (`route` = `summary`): `url`, `route`, `antiBotVendors` (union over routes), `checkedAt`, plus:

| Field | Type | Description |
|---|---|---|
| `cheapestWorkingRoute` | string or null | First route that worked, in the order direct, datacenter, googleSerp, residential. Null if none worked. |
| `workingRoutes` | array | Routes that returned `ok`. |
| `blockedRoutes` | array | Routes that were blocked. |
| `notApplicableRoutes` | array | Selected routes that don't apply (Google SERP for URLs other than Google Search, or rejected by the SERP proxy). |
| `estimatedCostPer1000Pages` | number or null | Rough proxy cost in USD for 1,000 pages like this one on the cheapest working route. Direct and datacenter are $0. Google SERP is $2.50. Residential is an **estimate**: `responseBytes` × $8 per GB (Apify's residential list price) × 1,000. It excludes compute and this Actor's own price. |
| `costBasis` | string or null | How the estimate was calculated. |
| `verdict` | string | One-sentence answer. |

### How much does it cost?

This Actor uses **pay-per-event** pricing. You pay for results, not for compute time:

| Event | Price | When |
|---|---|---|
| Result row | $0.0015 | Each row in the dataset: one per URL and tested route, plus one summary row per URL. |
| Residential proxy check | $0.015 | Extra charge, on top of the result row, for each request through the residential proxy that gets a response from the site (a page or a block page), including retries. Not charged on timeouts, connection errors or proxy errors. |
| Google SERP proxy check | $0.006 | Extra charge, on top of the result row, for each request through the Google SERP proxy that gets a response from Google, including retries. Not charged on timeouts, connection errors or URLs the proxy rejects. |

Examples:

- 2 URLs × direct + datacenter = 6 rows = **$0.009**.
- 100 URLs × direct + datacenter = 300 rows = **$0.45**.
- 10 URLs × direct + datacenter + residential, 1 attempt = 40 rows + 10 residential checks = **$0.21**.

Invalid URLs, hosts that don't resolve, and routes that don't apply are never charged. Set **Maximum cost per run** in the run options to cap spending. Before starting a URL, the Actor sets aside that URL's worst-case cost. URLs that don't fit wait for running ones to finish. When nothing is running and the next URL still doesn't fit, the Actor stops cleanly and lists the unchecked URLs in `FAILED_INPUTS`. Paid proxy checks already made are not refunded, even if a run is aborted before that URL's rows are saved.

### Tips

- Start with `direct` + `datacenter`. Add `residential` only for URLs where both are blocked.
- Use 2-3 **attempts per route** when results look random. Some vendors block only part of the IP pool.
- A **marker** (a product name, a price CSS class, `"__NEXT_DATA__"`) shows that the real content arrived, not just a 200 shell.
- Try `safari` or `firefox` fingerprints. Some sites score TLS fingerprints differently.
- `hasNextData` or `hasJsonLd` = true often means the data can be parsed from embedded JSON instead of HTML.
- The checker does **not** run JavaScript. A site that works only after a JS challenge shows as blocked here, even though a real browser might get through.

### FAQ

**Is it legal to use?**
The Actor fetches only publicly available pages, the same way a browser does. It sends one request per route (plus the retries you allow), so the load on any site is tiny. It does not log in, solve CAPTCHAs or bypass any protection. It measures what each route receives. Follow the target site's terms of service and your local law, and get legal advice if you are unsure.

**Does it solve CAPTCHAs or bypass bot protection?**
No. It reports whether a challenge appeared and who served it. It doesn't try to solve it.

**Does it store page content or personal data?**
No. Page bodies are never stored. Only facts about the response are saved: status, size, title, vendor fingerprints and marker matches. Cookie values are not recorded either.

**Why is the residential check more expensive?**
Residential proxy traffic is billed to the Actor per GB of bandwidth, and each check is a full page download. The download stops at 3 MB, so one check never moves more than about 3 MB. The flat price per answered request covers that cost whether the site answers with a page or a block page.

**How accurate is the cost estimate?**
It is a rough guide. It uses the decompressed page size and Apify's residential list price of $8 per GB, so it usually overestimates: proxies carry compressed bytes. It ignores the sub-resources (images, scripts) a browser would load, and it excludes compute and this Actor's price.

**Why does a site work on direct but show a vendor in `antiBotVendors`?**
Many sites load a bot-protection script or cookie on every page but only challenge suspicious traffic. "Present" and "blocked" are reported separately.

**Why are results different between runs?**
Proxy IPs rotate, and vendors score each IP and fingerprint separately. Run with more attempts, or schedule repeated checks, to see the pattern.

### Support

Found a site that's misclassified, or want another vendor detected? Open an issue in the **Issues** tab with the URL and the run ID.

# Actor input Schema

## `startUrls` (type: `array`):

Pages to test, up to 100 per run (extra URLs are listed in FAILED_INPUTS as skipped). Enter full URLs starting with http:// or https://. Private, loopback and metadata addresses are rejected. Each URL gets one request per selected route (more if you raise attempts).

## `routes` (type: `array`):

Access routes to try for every URL. Direct = the Actor's own server IP. Datacenter = Apify datacenter proxy. Residential = Apify residential proxy (extra charge per request that gets a response). Google SERP = Apify Google SERP proxy, only for Google Search URLs like www.google.com/search?q=... (extra charge per request that gets a response; other URLs are marked not applicable and not charged).

## `residentialCountry` (type: `string`):

Two-letter country code (ISO 3166-1) for the residential route, e.g. `US`, `GB`, `DE`. Ignored by other routes.

## `browserFingerprint` (type: `string`):

Which browser's TLS and HTTP/2 fingerprint and default headers each request copies. No JavaScript is executed.

## `markers` (type: `array`):

Optional case-sensitive strings to look for in each response body (for example a price class or a product name), to confirm you got the real page and not a stub. Results appear in `markersFound`.

## `attemptsPerRoute` (type: `integer`):

How many times to try a route when it is blocked or times out. Each proxy attempt uses a new proxy IP; direct attempts always use the Actor's server IP. A route stops at its first working response or at a settled answer such as 404.

## `timeoutSecs` (type: `integer`):

Maximum time for one request, including the download. Google SERP answers can take 10-20 seconds.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://example.com"
    },
    {
      "url": "https://www.wikipedia.org"
    }
  ],
  "routes": [
    "direct",
    "datacenter"
  ],
  "residentialCountry": "US",
  "browserFingerprint": "chrome",
  "markers": [],
  "attemptsPerRoute": 1,
  "timeoutSecs": 20
}
```

# Actor output Schema

## `summary` (type: `string`):

No description

## `routeResults` (type: `string`):

No description

## `failedInputs` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://example.com"
        },
        {
            "url": "https://www.wikipedia.org"
        }
    ],
    "routes": [
        "direct",
        "datacenter"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("hebaabdalaziz/anti-bot-proxy-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        { "url": "https://example.com" },
        { "url": "https://www.wikipedia.org" },
    ],
    "routes": [
        "direct",
        "datacenter",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("hebaabdalaziz/anti-bot-proxy-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://example.com"
    },
    {
      "url": "https://www.wikipedia.org"
    }
  ],
  "routes": [
    "direct",
    "datacenter"
  ]
}' |
apify call hebaabdalaziz/anti-bot-proxy-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,hebaabdalaziz/anti-bot-proxy-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/1DTeXknMQGEVvT5Zh/builds/ooH4mo7vbzlEjr71D/openapi.json