# Crawl Integrity Verifier (Anti-Cloaking) (`ernestmarza/crawl-integrity-verifier`) Actor Fetches the same URL under several identities - real browser, declared AI crawlers, anonymous client, datacenter proxy - and returns a semantic divergence score, a cloaking verdict and the diff. Filters timestamps, session ids, nonces, ads and element reordering so only meaningful differences count. - **URL**: https://apify.com/ernestmarza/crawl-integrity-verifier.md - **Developed by:** [Ernest MarzΓ‘](https://apify.com/ernestmarza) (community) - **Categories:** Integrations, Other - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1,500.00 / 1,000 url verifieds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Crawl Integrity Verifier (Anti-Cloaking) **Is the page your agent just read the same page a human would see?** An Apify Actor. Give it a URL; it fetches that URL several times under different identities - a real browser, ClaudeBot, GPTBot, an anonymous client, a datacenter proxy - and tells you how much the content actually differs, with the diff. Output is a `divergenceScore` from 0 to 1, a `verdict` of `clean`, `suspicious` or `cloaked`, and the exact text that changed between which routes. ### The problem As crawler verification becomes real - Web Bot Auth and the signature schemes around it - a server can be certain which agent is calling. That certainty is useful. It is also the ability to answer that agent differently, and nothing in the protocol obliges anyone to disclose that they have. The complaint, from a Hacker News thread on crawler verification (item 47339253, 11 March 2026): > sites will start serving different content to verified crawlers vs real users... > you have no guarantee it matches what a human sees, and that data quality problem > won't surface until your agent starts acting on selectively curated information The failure is silent by construction. Both real cases found while building this Actor returned **HTTP 200 and 202**. A crawler that checks `response.ok` sees success in both, ingests a challenge page or a curated page as though it were the article, and has no local evidence that anything is wrong. The problem does not surface until an agent acts on it. ### What it actually found These are measurements from this Actor's own normalizer, run against live sites on 27 July 2026. They are reported as they came out. #### The noise floor is genuinely near zero Fourteen pages that returned a usable browser response were each fetched twice under a **byte-identical** browser identity and scored against themselves: | | median | mean | p90 | worst | |---|---|---|---|---| | Semantic score (this Actor) | 0.000 | 0.0003 | 0.000 | 0.0049 | | Naive HTML diff, same pairs | 0.000 | 0.0013 | 0.0075 | 0.0098 | That worst case of 0.005 is what the alarm thresholds are calibrated against. #### Declared AI crawlers are mostly blocked, not lied to Across the same fourteen pages: | Route | Served 200 | Refused | Divergence from browser (max) | |---|---|---|---| | ClaudeBot | 9 | 5 (three 403, two 406) | **0.000** | | GPTBot | 12 | 2 (406) | **0.000** | | Googlebot | 13 | 1 (403) | **1.000** on one URL | | No user agent | 14 | 0 | 0.000 | | Automation tells (headless) | 14 | 0 | 0.042 | **The headline finding: the dominant response to a declared AI crawler is refusal, not curation.** Where ClaudeBot and GPTBot were served at all, they were served byte-equivalent content every single time. If you expected widespread AI-specific cloaking today, the data does not support it. What it supports is that the anxiety is well-founded but early: the mechanism exists and is in use, the targeting is not yet aimed at AI crawlers specifically. An earlier round against 32 homepages found the same pattern: 20 usable references, ClaudeBot served on 11 with zero divergence on all 11. #### But cloaking is real, and it is invisible Two sites out of roughly 46 distinct domains probed - call it 4% - did serve materially different content by identity. Both returned a 2xx status to every route. **`link.springer.com`**, an academic article. Divergence **1.000**. | Identity | Status | Bytes | Readable content | Title | |---|---|---|---|---| | Real browser | **200** | 3,036 | 209 chars | "Client Challenge" | | No user agent | **200** | 3,036 | 209 chars | "Client Challenge" | | Googlebot | **200** | 1,145,347 | **256,401 chars** | the actual paper | **`booking.com`**, the homepage. Divergence **1.000**. | Identity | Status | Bytes | Readable content | |---|---|---|---| | Real browser | **202** | 7,033 | **0 chars** | | ClaudeBot | **202** | 3,962 | **0 chars** | | GPTBot | **200** | 439,321 | 4,020 chars, 239 blocks | In both cases a crawler checking only the status code records a success and stores a challenge page as content. That is the entire thesis, reproduced twice. Note the direction: in both, the *browser* was the identity being starved. Whether you call that cloaking or aggressive bot mitigation, the operational consequence for anyone comparing agent output against "what a human sees" is identical, and neither response said so. #### Neither finding reproduces from Apify's own IPs Both cases above were found from a residential European home connection. Re-running the same URLs from the Apify platform, including through both the datacenter and the residential proxy, gives a completely different picture: **`link.springer.com` from Apify** - every route, including Googlebot, got the 209-character challenge page: | browser | control | googlebot | no-UA | datacenter | residential | |---|---|---|---|---|---| | 209 ch | 209 ch | 209 ch | 209 ch | 209 ch | 209 ch | **`booking.com` from Apify** - every route, including GPTBot, got an HTTP 202 with zero readable content. Verdict in both cases: `clean`, divergence `0.000`, **confidence `low`**, signal `reference-looks-challenged`. This is the correct behaviour and the guard rails did their job - the Actor did not issue a confident clean bill of health on a page it never actually saw - but the implication is blunt and belongs at the top of anyone's expectations: > **The network vantage point dominates the result.** A site that differentiates by > identity for one observer may refuse every identity from another. A `clean` verdict > with `confidence: low` and a `reference-looks-challenged` signal means "I could not > see the page", not "the page is fine". Read the confidence field before believing > a clean result. Where this Actor runs is therefore part of the measurement, not an implementation detail. Running it from Apify's infrastructure is the convenient option; running the same check from the network your agent actually crawls from is the meaningful one. #### Why the normalization is the product A diff that does not understand noise cannot tell these apart. Measured on a synthetic page where every volatile element differs - clock, session id, CSRF nonce, rotating ad, view counter, comment order - versus the same page with four fifths of its body withheld: | | Naive HTML diff | This Actor | |---|---|---| | Same page, all noise varied | **0.344** | **0.000** | | Cloaked page, content withheld | 0.532 | 0.529 | The naive diff reports 0.344 for noise and 0.532 for cloaking. Those overlap; no threshold separates them. After normalization the same two cases are 0.000 and 0.529, and any threshold between them works. **Without the normalizer there is no product**, and this is the number that says so. On live CDN-cached pages the naive diff also scores near zero, because such pages genuinely are static. The normalizer earns its keep on pages that are not: one challenge page in the sample scored **0.581 naive against 0.000 semantic** between two identical requests. ### How it works #### Routes Each route is one HTTP fetch of the same URL under a different identity. User-agent strings are the ones the operators publish, verbatim, because sites match on the exact token. | Route | What it isolates | |---|---| | `browser` | The reference. Chrome 131 with a complete navigation header set. Always on. | | `browser-control` | The same identity again. Its divergence is the page's own noise floor. | | `claudebot`, `gptbot`, `perplexitybot`, `meta-externalagent` | Declared AI crawler identities | | `googlebot` | Search indexing, for contrast with the AI crawlers | | `no-user-agent` | No UA header at all | | `generic-client` | An honest script (curl) that is not a declared AI crawler | | `automation-tells` | Headless Chrome plus the headers an instrumented browser leaks | | `datacenter-proxy` | The reference identity from a datacenter IP | | `residential-proxy` | The reference identity from a residential IP. Optional, see below. | The **control route is the design decision that makes the rest work.** Scoring against zero means any page with a live element looks suspicious. Scoring against a second fetch of the identical identity means each page is judged against its own measured variability. Every reported `divergenceScore` is `max(0, rawScore - noiseFloor)`. Routes run sequentially with a pause. Firing six simultaneous requests at one URL is itself an automation signal and would change the thing being measured. #### The normalizer A diff between two fetches of any real page finds dozens of differences that mean nothing. All of the following are removed or neutralised before anything is compared: - **Machinery**: `script`, `style`, `noscript`, `svg`, `iframe`, `template`, hidden elements, `aria-hidden` subtrees. - **Rotating containers**: ad slots, carousels, "trending now" rails, recommendation blocks, cookie banners, chat widgets - matched on class, id, `data-testid` and role, with separator-anchored patterns so that `ad` does not match `header`, `read-more` or `download`. - **Volatile tokens**, replaced with stable placeholders: ISO and HTTP timestamps, named and numeric dates, clock times, relative times in four languages, UUIDs, hex request ids and short commit hashes, high-entropy session tokens and nonces, thousands-separated and abbreviated counters, copyright years. - **Typography**: non-breaking spaces, zero-width characters, smart quotes, dashes. - **Link junk**: `utm_*`, click ids, session parameters, cache-busters; fragments; parameter ordering; host case. - **Element order**: blocks are aligned as sets, so a reshuffled product grid or a differently sorted comment list scores zero. What survives is split into leaf text blocks, and blocks are aligned in three passes: exact hash match, then greedy fuzzy match on a blend of word-trigram and word-set overlap, then whatever is left is genuinely present on one side only. A block that fuzzy-matches is reported as an **edit** with both versions and a similarity, and contributes only its changed fraction to the score - a paragraph 90% the same has diverged by 10%, not by 100%. Everything is weighted by text length. This matters more than any other choice here: unweighted, a page that lost its entire article body and a page that lost a "Skip to content" link have diverged by the same amount, which is nonsense. **Prices bypass the noise pipeline entirely.** The counter rule that correctly turns `1,204 comments` into ` comments` would just as happily turn `$1,204.00` into `$` and destroy the single most valuable signal the Actor can produce. Prices are extracted first, parsed under both decimal conventions, and compared separately. #### Scoring ```` score = contentDivergence \+ 0.15 \* priceDivergence \+ 0.10 \* linkDivergence \+ 0.05 \* metadataDivergence (clamped to 1) ```` Content divergence *is* the score; the rest are bounded surcharges that can add at most 0.30. This is not a weighted average, and the difference is not cosmetic: under an average, content had to share its budget with prices and links, which meant a page withholding four fifths of its body - the commonest form of cloaking, with no price or link change to show for it - could not mathematically reach the alarm threshold. It scored 0.32 against a threshold of 0.35. A detector that cannot reach its own alarm on its primary target is not calibrated, it is broken. A confirmed same-currency price conflict floors the score at 0.75 regardless of everything else. There is no reading of "19.99 to a browser, 29.99 to an agent" in which the two identities were shown the same offer. | Verdict | Score | Rationale | |---|---|---| | `clean` | < 0.08 | Worst observed noise floor was 0.005; this is more than an order of magnitude above it | | `suspicious` | 0.08 - 0.35 | Beyond anything a page was observed to differ from itself | | `cloaked` | >= 0.35, or any price conflict | A meaningful fraction of the readable page differs | #### Blocking is not cloaking A 403 to ClaudeBot is refusal, not deception, and conflating the two would be the most damaging false positive this Actor could produce. Blocked routes are reported in `routesUnavailable` with the reason and are never folded into the score. Given that refusal turned out to be the dominant behaviour, this distinction does most of the work in keeping the verdicts honest. ### Input Minimal: ```json { "urls": [{ "url": "https://example.com/product/123" }] } ```` Full: ```json { "urls": [ { "url": "https://example.com/product/123" }, { "url": "https://example.com/article/456" } ], "routes": [ "browser", "browser-control", "claudebot", "gptbot", "googlebot", "no-user-agent", "automation-tells", "datacenter-proxy" ], "requestTimeoutSecs": 30, "delayBetweenRoutesMs": 500, "maxDiffsReported": 10, "includeBlockText": true, "failOnCloaked": false } ``` Verify **the exact page an agent will act on** - a product page, an article, a listing. Homepages are usually static CDN shells with nothing to differentiate, which is why the first measurement round found so little. ### Output Per URL, one dataset item: | Field | What it tells you | |---|---| | `verdict` | `clean`, `suspicious` or `cloaked` | | `divergenceScore` | 0 to 1, already net of this page's measured noise floor | | `confidence` | `high`, `medium` or `low`. Low when the reference page was too thin to judge | | `summary` | One paragraph in plain language | | `signals[]` | Named findings: `content-withheld-from-agent`, `price-conflict`, `reference-looks-challenged`, ... | | `routesCompared[]` | Every route with status, bytes, **characters of readable content**, and block count | | `routesUnavailable[]` | Every route that failed or was refused, with the reason | | `semanticDiffs[]` | Per route pair: the sub-scores, the changed text, price conflicts, link deltas | | `noiseFiltered` | How much variation was discarded, by rule, and the naive score for comparison | The `contentChars` column in `routesCompared` is where withheld content becomes obvious at a glance: 209 against 256,401 needs no further analysis. `noiseFiltered` ships because a user looking at a low score is entitled to know whether the score is low because the pages agree or because we filtered the disagreement away. The key-value store record `OUTPUT` holds a run-level summary: verdict counts, the worst divergence, the cloaked and suspicious URL lists. That is the object to read when chaining - it answers "can I trust what I just crawled" without parsing the dataset. ### Pricing Pay per event, one event: `url-verified`, **$1.50** per URL. The obvious move is to price this like a scraper, at a fraction of a cent per row, and the obvious move is wrong, because this Actor does not return rows. It returns a decision. The unit of value is one answered question, and the cost of getting that answer wrong is an agent transacting on a curated version of reality. **A URL is charged only when at least two routes actually returned a comparable page.** A URL where everything was blocked produced no verdict, and charging for it would be charging for our own failure to get an answer. ### Honest limitations Read this section before trusting a verdict. - **No JavaScript is executed.** Every route is a plain HTTP fetch. This is a deliberate trade: a headless browser would be more faithful but ten times slower, and would add rendering timing as a second source of variance to a measurement whose entire value is that its variance is understood. The consequence is that on a client-rendered site all routes see the same empty shell and the Actor will report `clean` with a `reference-looks-challenged` signal and low confidence. Read the confidence field. - **The browser route is not a human.** It sends browser headers over a plain HTTP client, so its TLS and HTTP/2 fingerprints do not match Chrome. Many sites detect this: in the sample, **18 of 32 URLs refused the browser route outright**. Where the reference route is challenged, the comparison is between two kinds of bot, not between a bot and a human. This is the single biggest weakness of the approach. - **Where you run it changes the answer.** See the section above: both confirmed cases vanish when the same URLs are checked from Apify's IPs, because from there every identity is challenged equally. This is the largest caveat on any result this Actor produces. - **Residential proxy is optional and off by default.** When it cannot be configured the route is dropped, the reason is recorded in `routesUnavailable`, and the run continues. Both paths are verified: the degradation path against a local run with no proxy credentials, and the working path on the platform, where the account used did have residential access. It did not help - residential and datacenter IPs got the same challenge page on both positive cases. - **The base rate is low.** Two differentiating sites out of roughly 46 domains probed, and both only from one vantage point. This tool is worth running on the pages an agent will transact on, not as a blanket sweep. - **A single page cannot prove intent.** The Actor reports what differs. Whether that difference is cloaking, bot mitigation, geo-personalisation or an A/B test is a judgement it does not make. - **The rotating-container list is heuristic.** A site using unusual class names for its ad slots will have them scored as content. The `noiseFiltered` counts are there so you can notice when nothing was dropped from a page that clearly has ads. - **Sample size.** The measurements above are 64 fetch-sets across two rounds on one afternoon from one network. They establish that the noise floor is near zero and that the phenomenon exists. They are not a survey of the web. ### Development ```bash npm install npm run build npm test # 36 tests npm run typecheck ``` The test suite is mostly about the normalizer, because that is where this breaks. The central pair of tests: two fetches of the same page with every volatile element different must score below 0.01, and the same page with its body withheld must score above the cloaking threshold. Both assert on the same fixture generator, so neither can be satisfied by weakening the other. Two bugs were found by those tests and fixed rather than asserted around: a fourteen-character hex request id slipping under a sixteen-character redaction threshold and putting a floor of 0.058 under every comparison, and a URL with a non-http scheme being silently coerced into an https URL with the wrong host. # Actor input Schema ## `urls` (type: `array`): The pages to check. Each one is fetched once per route and scored independently. Up to 200 per run. Verifying the exact page an agent will act on - a product page, an article, a listing - tells you far more than verifying a homepage, which is usually a static CDN shell with nothing to differentiate. ## `routes` (type: `array`): Which identities to fetch each URL under. 'browser' is always included: it is the reference every other route is measured against. Keep 'browser-control' enabled unless you have a reason not to - it is a second fetch under the identical browser identity, and its divergence is the page's own noise floor, which every other score is corrected against. Without it, scores are measured against zero and any page with a live element looks suspicious. ## `requestTimeoutSecs` (type: `integer`): How long to wait for each individual fetch before giving up on that route. ## `delayBetweenRoutesMs` (type: `integer`): Pause between the fetches for one URL. Routes run sequentially on purpose: firing six simultaneous requests at one URL is itself an automation signal and would change the thing being measured. Raise this on rate-limited sites. ## `maxDiffsReported` (type: `integer`): How many route comparisons to include in full in semanticDiffs, worst first. The verdict and the score always consider every route regardless of this setting. ## `includeBlockText` (type: `boolean`): Include the actual text that differs, truncated to 200 characters per block. Turn this off for a smaller, quotation-free result when you only need the score and the verdict. ## `failOnCloaked` (type: `boolean`): Exit with a failed run when any URL returns a 'cloaked' verdict. For wiring this into a pipeline that should stop rather than proceed on data it cannot trust. Results are still written in full before the run fails. ## Actor input object example ```json { "urls": [ { "url": "https://example.com/" } ], "routes": [ "browser", "browser-control", "claudebot", "gptbot", "no-user-agent", "automation-tells", "datacenter-proxy" ], "requestTimeoutSecs": 30, "delayBetweenRoutesMs": 500, "maxDiffsReported": 10, "includeBlockText": true, "failOnCloaked": false } ``` # Actor output Schema ## `summary` (type: `string`): A single JSON object for the whole run: how many URLs came back clean, suspicious or cloaked, the worst divergence score observed, the list of cloaked and suspicious URLs, and which routes were unavailable and why. This is the object to read when chaining: it answers 'can I trust what I just crawled' without parsing the dataset. ## `verdicts` (type: `string`): One dataset item per URL, each containing verdict (clean, suspicious or cloaked), divergenceScore from 0 to 1, confidence, a plain-language summary, the named signals that fired, every route that was compared with its status and content size, every route that was unavailable with the reason, the semantic diffs between route pairs including the differing text and any price conflicts, and the noiseFiltered report showing how much variation was discarded as meaningless. ## `cloakedOnly` (type: `string`): The same dataset filtered to the URLs that failed. Fetch this when you only want to know what to throw away. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ { "url": "https://example.com/" } ] }; // Run the Actor and wait for it to finish const run = await client.actor("ernestmarza/crawl-integrity-verifier").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": [{ "url": "https://example.com/" }] } # Run the Actor and wait for it to finish run = client.actor("ernestmarza/crawl-integrity-verifier").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ { "url": "https://example.com/" } ] }' | apify call ernestmarza/crawl-integrity-verifier --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=ernestmarza/crawl-integrity-verifier", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Crawl Integrity Verifier (Anti-Cloaking)", "description": "Fetches the same URL under several identities - real browser, declared AI crawlers, anonymous client, datacenter proxy - and returns a semantic divergence score, a cloaking verdict and the diff. Filters timestamps, session ids, nonces, ads and element reordering so only meaningful differences count.", "version": "0.1", "x-build-id": "F46Qvki5Wx1TfC9Ip" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/ernestmarza~crawl-integrity-verifier/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-ernestmarza-crawl-integrity-verifier", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/ernestmarza~crawl-integrity-verifier/runs": { "post": { "operationId": "runs-sync-ernestmarza-crawl-integrity-verifier", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/ernestmarza~crawl-integrity-verifier/run-sync": { "post": { "operationId": "run-sync-ernestmarza-crawl-integrity-verifier", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "urls" ], "properties": { "urls": { "title": "URLs to verify", "type": "array", "description": "The pages to check. Each one is fetched once per route and scored independently. Up to 200 per run. Verifying the exact page an agent will act on - a product page, an article, a listing - tells you far more than verifying a homepage, which is usually a static CDN shell with nothing to differentiate.", "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "routes": { "title": "Routes (identities to compare)", "uniqueItems": true, "type": "array", "description": "Which identities to fetch each URL under. 'browser' is always included: it is the reference every other route is measured against. Keep 'browser-control' enabled unless you have a reason not to - it is a second fetch under the identical browser identity, and its divergence is the page's own noise floor, which every other score is corrected against. Without it, scores are measured against zero and any page with a live element looks suspicious.", "items": { "type": "string", "enum": [ "browser", "browser-control", "claudebot", "gptbot", "perplexitybot", "meta-externalagent", "googlebot", "no-user-agent", "generic-client", "automation-tells", "datacenter-proxy", "residential-proxy" ], "enumTitles": [ "Real browser (reference, always on)", "Real browser again (control, measures the noise floor)", "ClaudeBot", "GPTBot", "PerplexityBot", "Meta external agent", "Googlebot", "No user agent", "Generic HTTP client (curl)", "Browser with automation tells (headless)", "Browser via datacenter proxy", "Browser via residential proxy (paid plans only)" ] }, "default": [ "browser", "browser-control", "claudebot", "gptbot", "no-user-agent", "automation-tells", "datacenter-proxy" ] }, "requestTimeoutSecs": { "title": "Request timeout (seconds)", "minimum": 5, "maximum": 120, "type": "integer", "description": "How long to wait for each individual fetch before giving up on that route.", "default": 30 }, "delayBetweenRoutesMs": { "title": "Delay between routes (ms)", "minimum": 0, "maximum": 10000, "type": "integer", "description": "Pause between the fetches for one URL. Routes run sequentially on purpose: firing six simultaneous requests at one URL is itself an automation signal and would change the thing being measured. Raise this on rate-limited sites.", "default": 500 }, "maxDiffsReported": { "title": "Maximum route pairs detailed per URL", "minimum": 1, "maximum": 20, "type": "integer", "description": "How many route comparisons to include in full in semanticDiffs, worst first. The verdict and the score always consider every route regardless of this setting.", "default": 10 }, "includeBlockText": { "title": "Include the text of differing blocks", "type": "boolean", "description": "Include the actual text that differs, truncated to 200 characters per block. Turn this off for a smaller, quotation-free result when you only need the score and the verdict.", "default": true }, "failOnCloaked": { "title": "Fail the run if any URL is cloaked", "type": "boolean", "description": "Exit with a failed run when any URL returns a 'cloaked' verdict. For wiring this into a pipeline that should stop rather than proceed on data it cannot trust. Results are still written in full before the run fails.", "default": false } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```