Anti-Bot & Proxy Tier Checker

What does it do?

Anti-Bot & Proxy Tier Checker sends one real browser-like request to each URL through each access route you pick, then tells you which routes get the page:

• Direct: the Actor's own server IP, no proxy.
• Datacenter: Apify datacenter proxy.
• Residential: Apify residential proxy, in a country you choose.
• Google SERP: Apify Google SERP proxy, for Google Search URLs only (www.google.<tld>/search?...).

For every URL and route you get the HTTP status and whether the page really worked or was a challenge page. You also get which anti-bot vendor is in front of the site (Cloudflare, DataDome, Akamai, PerimeterX, AWS WAF, Incapsula, Kasada, Google's own wall), plus the response size and speed. A summary row per URL names the cheapest route that works and estimates its proxy cost per 1,000 pages.

Why use it?

• Scraper developers: pick the right proxy tier before writing a scraper. Don't pay for residential traffic when datacenter IPs work, and don't fight a DataDome wall with datacenter IPs.
• Budgeting: get a rough proxy cost per 1,000 pages from the real page size.
• SEO and site owners: check what bot protection your site (or a competitor's) shows to cloud and residential IPs, and whether real pages or challenge pages are served.
• Monitoring: schedule it to notice when a target adds or changes bot protection.

How to use it

1. Paste one or more full URLs, starting with http:// or https://, into URLs to check (up to 100 per run).
2. Pick the Routes to test. Direct and datacenter are cheapest; add residential or Google SERP when you need them.
3. Optionally set the residential country, browser fingerprint, text markers, attempts per route and timeout.
4. Click Start. Most runs finish in seconds.
5. Open the Summary per URL view for the answer, or Results per route for the details. Download as JSON, CSV or Excel, or read the dataset through the API.

Input

Field	Type	Default	Description
startUrls	array of URLs	(required)	Pages to test, max 100 per run. Extra URLs are listed in FAILED_INPUTS as skipped.
routes	array	["direct", "datacenter"]	Any of direct, datacenter, residential, googleSerp.
residentialCountry	string	US	Two-letter country code for the residential route.
browserFingerprint	string	chrome	chrome, safari, firefox or edge: the TLS/HTTP2 fingerprint and default headers each request copies.
markers	array of strings	[]	Case-sensitive strings to look for in each body. The results go in markersFound.
attemptsPerRoute	integer 1-3	1	Retries for a blocked or timed-out route. Each proxy attempt uses a new proxy IP; direct attempts always come from the Actor's server IP.
timeoutSecs	integer 5-30	20	Max time per request, for every route. Google SERP answers can take 10-20 s, so keep 20 or more for that route.

{
    "startUrls": [
        { "url": "https://www.bayt.com/en/uae/jobs/" },
        { "url": "https://www.google.com/search?q=web+scraping&hl=en&gl=us" }
    ],
    "routes": ["direct", "datacenter", "residential", "googleSerp"],
    "residentialCountry": "US",
    "browserFingerprint": "chrome",
    "markers": ["</html>"],
    "attemptsPerRoute": 1,
    "timeoutSecs": 20
}

The Google SERP route only applies to Google Search URLs: google.<tld> or www.google.<tld> with a /search path. For other URLs, including news.google.com, it is listed under notApplicableRoutes in the summary row. It is not tested and not charged. The same applies when the SERP proxy itself rejects a URL with HTTP 400.

Private, loopback, link-local and cloud-metadata addresses (for example 10.x, 192.168.x, 127.0.0.1, 169.254.169.254, localhost) are rejected.

Output

The dataset has one row per URL and route, plus one "route": "summary" row per URL. Below are real rows from a cloud run on 2026-10-12 for a Cloudflare-protected job board.

[
  {
    "url": "https://www.bayt.com/en/uae/jobs/",
    "route": "datacenter",
    "status": 403,
    "ok": false,
    "blocked": true,
    "blockReason": "403 Cloudflare challenge",
    "antiBotVendors": ["cloudflare"],
    "responseBytes": 5972,
    "bodyTruncated": false,
    "responseSecs": 0.64,
    "finalUrl": "https://www.bayt.com/en/uae/jobs/",
    "pageTitle": "Just a moment...",
    "hasNextData": false,
    "hasJsonLd": false,
    "proxyCountry": null,
    "fingerprint": "chrome",
    "attempts": 2,
    "checkedAt": "2026-10-12T13:13:58+00:00"
  },
  {
    "url": "https://www.bayt.com/en/uae/jobs/",
    "route": "residential",
    "status": 200,
    "ok": true,
    "blocked": false,
    "blockReason": null,
    "antiBotVendors": ["cloudflare"],
    "responseBytes": 247781,
    "bodyTruncated": false,
    "responseSecs": 1.03,
    "finalUrl": "https://www.bayt.com/en/uae/jobs/",
    "pageTitle": "Jobs in UAE (Oct 2026) - Bayt.com",
    "hasNextData": false,
    "hasJsonLd": true,
    "proxyCountry": "US",
    "fingerprint": "chrome",
    "attempts": 1,
    "checkedAt": "2026-10-12T13:13:58+00:00"
  },
  {
    "url": "https://www.bayt.com/en/uae/jobs/",
    "route": "summary",
    "cheapestWorkingRoute": "residential",
    "workingRoutes": ["residential"],
    "blockedRoutes": ["direct", "datacenter"],
    "notApplicableRoutes": [],
    "antiBotVendors": ["cloudflare"],
    "estimatedCostPer1000Pages": 1.9822,
    "costBasis": "Estimate: response size x $8 per GB (Apify residential proxy list price) x 1,000 pages.",
    "verdict": "Works on residential; cheapest working route: residential. Blocked on direct, datacenter.",
    "checkedAt": "2026-10-12T13:13:58+00:00"
  }
]

Other answers from the same test runs:

Site	Direct	Datacenter	Residential	Detected
eBay home	403 Forbidden (behind Akamai)	403 Forbidden (behind Akamai)	200 ok	akamai
G2	403 DataDome challenge	403 DataDome challenge	403 DataDome challenge	datadome, cloudflare
Trustpilot review page	403 AWS WAF challenge	403 AWS WAF challenge	403 AWS WAF challenge	aws-waf
Hyatt	429 Kasada challenge	429 Kasada challenge	429 Kasada challenge	kasada
Zillow home	200 ok	200 ok	200 ok	perimeterx
Google search	200 Google JavaScript wall	(not run)	(not run)	google; the Google SERP route returned results

URLs that are invalid, do not resolve, or were not checked are listed with a reason in the FAILED_INPUTS record of the run's key-value store. Nothing is charged for them.

Data fields

Fields on every route row (route = direct, datacenter, residential or googleSerp):

Field	Type	Description
url	string	Input URL, normalized.
route	string	Route tested.
status	integer or null	HTTP status of the reported attempt. Null when no response arrived. Codes 590-599 come from the Apify proxy: it could not reach the site.
ok	boolean	2xx and not a challenge or block page.
blocked	boolean	Refused: challenge or block page, block status (403, 429, 451, 503), timeout or dropped connection. A 401, 404, 405 or 500 is a settled answer, not a block, and is not retried.
blockReason	string or null	Why the page was not usable, e.g. 403 Cloudflare challenge, 429 Kasada challenge, 200 Google JavaScript wall, timeout, body timeout (headers received, body stalled), 404 Not Found. A vendor is named as the blocker only when its challenge or block page (or a cf-mitigated header) was seen; otherwise you get e.g. 403 Forbidden (behind Cloudflare). Null when ok.
antiBotVendors	array	Vendors fingerprinted in headers, cookie names or body: datadome, perimeterx, kasada, aws-waf, incapsula, akamai, cloudflare, google. A vendor can be present without blocking.
responseBytes	integer or null	Decompressed body size, capped at 3 MB.
bodyTruncated	boolean	True when the download was stopped early: the body passed 3 MB, or the timeout hit after the headers had arrived.
responseSecs	number or null	Seconds until the download finished, or until the error.
finalUrl	string or null	URL after redirects.
pageTitle	string or null	The page <title>.
hasNextData	boolean or null	Page contains Next.js __NEXT_DATA__ JSON.
hasJsonLd	boolean or null	Page contains application/ld+json structured data.
markersFound	object or null	Only when markers were given: {marker: true/false}.
proxyCountry	string or null	Requested residential country (residential rows only).
fingerprint	string	Browser fingerprint used.
attempts	integer	Requests made on this route. Proxy attempts each used a new proxy IP.
checkedAt	string	ISO 8601 time in UTC.

Fields on every summary row (route = summary): url, route, antiBotVendors (union over routes), checkedAt, plus:

Field	Type	Description
cheapestWorkingRoute	string or null	First route that worked, in the order direct, datacenter, googleSerp, residential. Null if none worked.
workingRoutes	array	Routes that returned ok.
blockedRoutes	array	Routes that were blocked.
notApplicableRoutes	array	Selected routes that don't apply (Google SERP for URLs other than Google Search, or rejected by the SERP proxy).
estimatedCostPer1000Pages	number or null	Rough proxy cost in USD for 1,000 pages like this one on the cheapest working route. Direct and datacenter are $0. Google SERP is $2.50. Residential is an estimate: responseBytes × $8 per GB (Apify's residential list price) × 1,000. It excludes compute and this Actor's own price.
costBasis	string or null	How the estimate was calculated.
verdict	string	One-sentence answer.

How much does it cost?

This Actor uses pay-per-event pricing. You pay for results, not for compute time:

Event	Price	When
Result row	$0.0015	Each row in the dataset: one per URL and tested route, plus one summary row per URL.
Residential proxy check	$0.015	Extra charge, on top of the result row, for each request through the residential proxy that gets a response from the site (a page or a block page), including retries. Not charged on timeouts, connection errors or proxy errors.
Google SERP proxy check	$0.006	Extra charge, on top of the result row, for each request through the Google SERP proxy that gets a response from Google, including retries. Not charged on timeouts, connection errors or URLs the proxy rejects.

Examples:

• 2 URLs × direct + datacenter = 6 rows = $0.009.
• 100 URLs × direct + datacenter = 300 rows = $0.45.
• 10 URLs × direct + datacenter + residential, 1 attempt = 40 rows + 10 residential checks = $0.21.

Invalid URLs, hosts that don't resolve, and routes that don't apply are never charged. Set Maximum cost per run in the run options to cap spending. Before starting a URL, the Actor sets aside that URL's worst-case cost. URLs that don't fit wait for running ones to finish. When nothing is running and the next URL still doesn't fit, the Actor stops cleanly and lists the unchecked URLs in FAILED_INPUTS. Paid proxy checks already made are not refunded, even if a run is aborted before that URL's rows are saved.

Tips

• Start with direct + datacenter. Add residential only for URLs where both are blocked.
• Use 2-3 attempts per route when results look random. Some vendors block only part of the IP pool.
• A marker (a product name, a price CSS class, "__NEXT_DATA__") shows that the real content arrived, not just a 200 shell.
• Try safari or firefox fingerprints. Some sites score TLS fingerprints differently.
• hasNextData or hasJsonLd = true often means the data can be parsed from embedded JSON instead of HTML.
• The checker does not run JavaScript. A site that works only after a JS challenge shows as blocked here, even though a real browser might get through.

FAQ

Is it legal to use? The Actor fetches only publicly available pages, the same way a browser does. It sends one request per route (plus the retries you allow), so the load on any site is tiny. It does not log in, solve CAPTCHAs or bypass any protection. It measures what each route receives. Follow the target site's terms of service and your local law, and get legal advice if you are unsure.

Does it solve CAPTCHAs or bypass bot protection? No. It reports whether a challenge appeared and who served it. It doesn't try to solve it.

Does it store page content or personal data? No. Page bodies are never stored. Only facts about the response are saved: status, size, title, vendor fingerprints and marker matches. Cookie values are not recorded either.

Why is the residential check more expensive? Residential proxy traffic is billed to the Actor per GB of bandwidth, and each check is a full page download. The download stops at 3 MB, so one check never moves more than about 3 MB. The flat price per answered request covers that cost whether the site answers with a page or a block page.

How accurate is the cost estimate? It is a rough guide. It uses the decompressed page size and Apify's residential list price of $8 per GB, so it usually overestimates: proxies carry compressed bytes. It ignores the sub-resources (images, scripts) a browser would load, and it excludes compute and this Actor's price.

Why does a site work on direct but show a vendor in antiBotVendors? Many sites load a bot-protection script or cookie on every page but only challenge suspicious traffic. "Present" and "blocked" are reported separately.

Why are results different between runs? Proxy IPs rotate, and vendors score each IP and fingerprint separately. Run with more attempts, or schedule repeated checks, to see the pattern.

Support

Found a site that's misclassified, or want another vendor detected? Open an issue in the Issues tab with the URL and the run ID.