Crawl Integrity Verifier (Anti-Cloaking)

Is the page your agent just read the same page a human would see?

An Apify Actor. Give it a URL; it fetches that URL several times under different identities - a real browser, ClaudeBot, GPTBot, an anonymous client, a datacenter proxy - and tells you how much the content actually differs, with the diff.

Output is a divergenceScore from 0 to 1, a verdict of clean, suspicious or cloaked, and the exact text that changed between which routes.

The problem

As crawler verification becomes real - Web Bot Auth and the signature schemes around it - a server can be certain which agent is calling. That certainty is useful. It is also the ability to answer that agent differently, and nothing in the protocol obliges anyone to disclose that they have.

The complaint, from a Hacker News thread on crawler verification (item 47339253, 11 March 2026):

sites will start serving different content to verified crawlers vs real users... you have no guarantee it matches what a human sees, and that data quality problem won't surface until your agent starts acting on selectively curated information

The failure is silent by construction. Both real cases found while building this Actor returned HTTP 200 and 202. A crawler that checks response.ok sees success in both, ingests a challenge page or a curated page as though it were the article, and has no local evidence that anything is wrong. The problem does not surface until an agent acts on it.

What it actually found

These are measurements from this Actor's own normalizer, run against live sites on 27 July 2026. They are reported as they came out.

The noise floor is genuinely near zero

Fourteen pages that returned a usable browser response were each fetched twice under a byte-identical browser identity and scored against themselves:

	median	mean	p90	worst
Semantic score (this Actor)	0.000	0.0003	0.000	0.0049
Naive HTML diff, same pairs	0.000	0.0013	0.0075	0.0098

That worst case of 0.005 is what the alarm thresholds are calibrated against.

Declared AI crawlers are mostly blocked, not lied to

Across the same fourteen pages:

Route	Served 200	Refused	Divergence from browser (max)
ClaudeBot	9	5 (three 403, two 406)	0.000
GPTBot	12	2 (406)	0.000
Googlebot	13	1 (403)	1.000 on one URL
No user agent	14	0	0.000
Automation tells (headless)	14	0	0.042

The headline finding: the dominant response to a declared AI crawler is refusal, not curation. Where ClaudeBot and GPTBot were served at all, they were served byte-equivalent content every single time. If you expected widespread AI-specific cloaking today, the data does not support it. What it supports is that the anxiety is well-founded but early: the mechanism exists and is in use, the targeting is not yet aimed at AI crawlers specifically.

An earlier round against 32 homepages found the same pattern: 20 usable references, ClaudeBot served on 11 with zero divergence on all 11.

But cloaking is real, and it is invisible

Two sites out of roughly 46 distinct domains probed - call it 4% - did serve materially different content by identity. Both returned a 2xx status to every route.

link.springer.com, an academic article. Divergence 1.000.

Identity	Status	Bytes	Readable content	Title
Real browser	200	3,036	209 chars	"Client Challenge"
No user agent	200	3,036	209 chars	"Client Challenge"
Googlebot	200	1,145,347	256,401 chars	the actual paper

booking.com, the homepage. Divergence 1.000.

Identity	Status	Bytes	Readable content
Real browser	202	7,033	0 chars
ClaudeBot	202	3,962	0 chars
GPTBot	200	439,321	4,020 chars, 239 blocks

In both cases a crawler checking only the status code records a success and stores a challenge page as content. That is the entire thesis, reproduced twice.

Note the direction: in both, the browser was the identity being starved. Whether you call that cloaking or aggressive bot mitigation, the operational consequence for anyone comparing agent output against "what a human sees" is identical, and neither response said so.

Neither finding reproduces from Apify's own IPs

Both cases above were found from a residential European home connection. Re-running the same URLs from the Apify platform, including through both the datacenter and the residential proxy, gives a completely different picture:

link.springer.com from Apify - every route, including Googlebot, got the 209-character challenge page:

browser	control	googlebot	no-UA	datacenter	residential
209 ch	209 ch	209 ch	209 ch	209 ch	209 ch

booking.com from Apify - every route, including GPTBot, got an HTTP 202 with zero readable content.

Verdict in both cases: clean, divergence 0.000, confidence low, signal reference-looks-challenged.

This is the correct behaviour and the guard rails did their job - the Actor did not issue a confident clean bill of health on a page it never actually saw - but the implication is blunt and belongs at the top of anyone's expectations:

The network vantage point dominates the result. A site that differentiates by identity for one observer may refuse every identity from another. A clean verdict with confidence: low and a reference-looks-challenged signal means "I could not see the page", not "the page is fine". Read the confidence field before believing a clean result.

Where this Actor runs is therefore part of the measurement, not an implementation detail. Running it from Apify's infrastructure is the convenient option; running the same check from the network your agent actually crawls from is the meaningful one.

Why the normalization is the product

A diff that does not understand noise cannot tell these apart. Measured on a synthetic page where every volatile element differs - clock, session id, CSRF nonce, rotating ad, view counter, comment order - versus the same page with four fifths of its body withheld:

	Naive HTML diff	This Actor
Same page, all noise varied	0.344	0.000
Cloaked page, content withheld	0.532	0.529

The naive diff reports 0.344 for noise and 0.532 for cloaking. Those overlap; no threshold separates them. After normalization the same two cases are 0.000 and 0.529, and any threshold between them works. Without the normalizer there is no product, and this is the number that says so.

On live CDN-cached pages the naive diff also scores near zero, because such pages genuinely are static. The normalizer earns its keep on pages that are not: one challenge page in the sample scored 0.581 naive against 0.000 semantic between two identical requests.

How it works

Routes

Each route is one HTTP fetch of the same URL under a different identity. User-agent strings are the ones the operators publish, verbatim, because sites match on the exact token.

Route	What it isolates
browser	The reference. Chrome 131 with a complete navigation header set. Always on.
browser-control	The same identity again. Its divergence is the page's own noise floor.
claudebot, gptbot, perplexitybot, meta-externalagent	Declared AI crawler identities
googlebot	Search indexing, for contrast with the AI crawlers
no-user-agent	No UA header at all
generic-client	An honest script (curl) that is not a declared AI crawler
automation-tells	Headless Chrome plus the headers an instrumented browser leaks
datacenter-proxy	The reference identity from a datacenter IP
residential-proxy	The reference identity from a residential IP. Optional, see below.

The control route is the design decision that makes the rest work. Scoring against zero means any page with a live element looks suspicious. Scoring against a second fetch of the identical identity means each page is judged against its own measured variability. Every reported divergenceScore is max(0, rawScore - noiseFloor).

Routes run sequentially with a pause. Firing six simultaneous requests at one URL is itself an automation signal and would change the thing being measured.

The normalizer

A diff between two fetches of any real page finds dozens of differences that mean nothing. All of the following are removed or neutralised before anything is compared:

• Machinery: script, style, noscript, svg, iframe, template, hidden elements, aria-hidden subtrees.
• Rotating containers: ad slots, carousels, "trending now" rails, recommendation blocks, cookie banners, chat widgets - matched on class, id, data-testid and role, with separator-anchored patterns so that ad does not match header, read-more or download.
• Volatile tokens, replaced with stable placeholders: ISO and HTTP timestamps, named and numeric dates, clock times, relative times in four languages, UUIDs, hex request ids and short commit hashes, high-entropy session tokens and nonces, thousands-separated and abbreviated counters, copyright years.
• Typography: non-breaking spaces, zero-width characters, smart quotes, dashes.
• Link junk: utm_*, click ids, session parameters, cache-busters; fragments; parameter ordering; host case.
• Element order: blocks are aligned as sets, so a reshuffled product grid or a differently sorted comment list scores zero.

What survives is split into leaf text blocks, and blocks are aligned in three passes: exact hash match, then greedy fuzzy match on a blend of word-trigram and word-set overlap, then whatever is left is genuinely present on one side only. A block that fuzzy-matches is reported as an edit with both versions and a similarity, and contributes only its changed fraction to the score - a paragraph 90% the same has diverged by 10%, not by 100%.

Everything is weighted by text length. This matters more than any other choice here: unweighted, a page that lost its entire article body and a page that lost a "Skip to content" link have diverged by the same amount, which is nonsense.

Prices bypass the noise pipeline entirely. The counter rule that correctly turns 1,204 comments into <NUM> comments would just as happily turn $1,204.00 into $<NUM> and destroy the single most valuable signal the Actor can produce. Prices are extracted first, parsed under both decimal conventions, and compared separately.

Scoring

score = contentDivergence
      + 0.15 * priceDivergence
      + 0.10 * linkDivergence
      + 0.05 * metadataDivergence          (clamped to 1)

Content divergence is the score; the rest are bounded surcharges that can add at most 0.30. This is not a weighted average, and the difference is not cosmetic: under an average, content had to share its budget with prices and links, which meant a page withholding four fifths of its body - the commonest form of cloaking, with no price or link change to show for it - could not mathematically reach the alarm threshold. It scored 0.32 against a threshold of 0.35. A detector that cannot reach its own alarm on its primary target is not calibrated, it is broken.

A confirmed same-currency price conflict floors the score at 0.75 regardless of everything else. There is no reading of "19.99 to a browser, 29.99 to an agent" in which the two identities were shown the same offer.

Verdict	Score	Rationale
clean	< 0.08	Worst observed noise floor was 0.005; this is more than an order of magnitude above it
suspicious	0.08 - 0.35	Beyond anything a page was observed to differ from itself
cloaked	>= 0.35, or any price conflict	A meaningful fraction of the readable page differs

Blocking is not cloaking

A 403 to ClaudeBot is refusal, not deception, and conflating the two would be the most damaging false positive this Actor could produce. Blocked routes are reported in routesUnavailable with the reason and are never folded into the score. Given that refusal turned out to be the dominant behaviour, this distinction does most of the work in keeping the verdicts honest.

Input

Minimal:

{ "urls": [{ "url": "https://example.com/product/123" }] }

Full:

{
  "urls": [
    { "url": "https://example.com/product/123" },
    { "url": "https://example.com/article/456" }
  ],
  "routes": [
    "browser", "browser-control", "claudebot", "gptbot",
    "googlebot", "no-user-agent", "automation-tells", "datacenter-proxy"
  ],
  "requestTimeoutSecs": 30,
  "delayBetweenRoutesMs": 500,
  "maxDiffsReported": 10,
  "includeBlockText": true,
  "failOnCloaked": false
}

Verify the exact page an agent will act on - a product page, an article, a listing. Homepages are usually static CDN shells with nothing to differentiate, which is why the first measurement round found so little.

Output

Per URL, one dataset item:

Field	What it tells you
verdict	clean, suspicious or cloaked
divergenceScore	0 to 1, already net of this page's measured noise floor
confidence	high, medium or low. Low when the reference page was too thin to judge
summary	One paragraph in plain language
signals[]	Named findings: content-withheld-from-agent, price-conflict, reference-looks-challenged, ...
routesCompared[]	Every route with status, bytes, characters of readable content, and block count
routesUnavailable[]	Every route that failed or was refused, with the reason
semanticDiffs[]	Per route pair: the sub-scores, the changed text, price conflicts, link deltas
noiseFiltered	How much variation was discarded, by rule, and the naive score for comparison

The contentChars column in routesCompared is where withheld content becomes obvious at a glance: 209 against 256,401 needs no further analysis.

noiseFiltered ships because a user looking at a low score is entitled to know whether the score is low because the pages agree or because we filtered the disagreement away.

The key-value store record OUTPUT holds a run-level summary: verdict counts, the worst divergence, the cloaked and suspicious URL lists. That is the object to read when chaining - it answers "can I trust what I just crawled" without parsing the dataset.

Pricing

Pay per event, one event: url-verified, $1.50 per URL.

The obvious move is to price this like a scraper, at a fraction of a cent per row, and the obvious move is wrong, because this Actor does not return rows. It returns a decision. The unit of value is one answered question, and the cost of getting that answer wrong is an agent transacting on a curated version of reality.

A URL is charged only when at least two routes actually returned a comparable page. A URL where everything was blocked produced no verdict, and charging for it would be charging for our own failure to get an answer.

Honest limitations

Read this section before trusting a verdict.

• No JavaScript is executed. Every route is a plain HTTP fetch. This is a deliberate trade: a headless browser would be more faithful but ten times slower, and would add rendering timing as a second source of variance to a measurement whose entire value is that its variance is understood. The consequence is that on a client-rendered site all routes see the same empty shell and the Actor will report clean with a reference-looks-challenged signal and low confidence. Read the confidence field.
• The browser route is not a human. It sends browser headers over a plain HTTP client, so its TLS and HTTP/2 fingerprints do not match Chrome. Many sites detect this: in the sample, 18 of 32 URLs refused the browser route outright. Where the reference route is challenged, the comparison is between two kinds of bot, not between a bot and a human. This is the single biggest weakness of the approach.
• Where you run it changes the answer. See the section above: both confirmed cases vanish when the same URLs are checked from Apify's IPs, because from there every identity is challenged equally. This is the largest caveat on any result this Actor produces.
• Residential proxy is optional and off by default. When it cannot be configured the route is dropped, the reason is recorded in routesUnavailable, and the run continues. Both paths are verified: the degradation path against a local run with no proxy credentials, and the working path on the platform, where the account used did have residential access. It did not help - residential and datacenter IPs got the same challenge page on both positive cases.
• The base rate is low. Two differentiating sites out of roughly 46 domains probed, and both only from one vantage point. This tool is worth running on the pages an agent will transact on, not as a blanket sweep.
• A single page cannot prove intent. The Actor reports what differs. Whether that difference is cloaking, bot mitigation, geo-personalisation or an A/B test is a judgement it does not make.
• The rotating-container list is heuristic. A site using unusual class names for its ad slots will have them scored as content. The noiseFiltered counts are there so you can notice when nothing was dropped from a page that clearly has ads.
• Sample size. The measurements above are 64 fetch-sets across two rounds on one afternoon from one network. They establish that the noise floor is near zero and that the phenomenon exists. They are not a survey of the web.

Development

npm install
npm run build
npm test          # 36 tests
npm run typecheck

The test suite is mostly about the normalizer, because that is where this breaks. The central pair of tests: two fetches of the same page with every volatile element different must score below 0.01, and the same page with its body withheld must score above the cloaking threshold. Both assert on the same fixture generator, so neither can be satisfied by weakening the other.

Two bugs were found by those tests and fixed rather than asserted around: a fourteen-character hex request id slipping under a sixteen-character redaction threshold and putting a floor of 0.058 under every comparison, and a URL with a non-http scheme being silently coerced into an https URL with the wrong host.