Scrape QA — Verify Any Scraper's Output Against Reality

Point it at a dataset from any scraper run and get back a trust report: which rows are verified against the live source pages, which are wrong, which couldn't be checked — and an overall trust score for the run.

Scrapers fail quietly. Stale results, missing fields, inconsistent runs — the same configuration returning different data twice, prices that changed since the crawl, values that were never on the page at all. You find out when a customer does, or you don't find out at all.

Scrape QA is an independent check: it re-fetches every source page itself and verifies your scraped values against what the live page actually says. Not schema validation. Not deduplication. Truth against reality.

The first dataset this actor ever QA'd was its own test author's: 20 rows scraped for the acceptance run, 3 corruptions planted deliberately. It caught both corrupted values outright — and the third, a fabricated field, exposed a blind spot that became a feature: fields outside the mapping are now explicitly reported as never-examined, because unexamined must not pass as verified. Then it flagged 2 more rows nobody planted: the test scraper had an HTML-entity bug (Shakespeare's Sonnets scraped as the literal title), and the verifier caught its own author's scraper shipping dirty data. That's the product.

What a verdict means

Every row gets one of:

• pass — full verification: we re-fetched the page, found your value, confirmed it was served live, and an independent LLM judge confirmed the value actually answers what was asked. ed25519-signed.
• fail — we checked, and the claim is wrong (value absent, or present as the wrong thing — a was-price, another variant, a related item).
• unverifiable — we could not independently confirm, by design: unreachable page, demonstrably stale cache, client-rendered content, or an uncheckable field. It is never a wrong answer — when we can check and the claim is wrong, we say fail.

The trust guarantee is structural, not statistical. A claim is never certified unless ScrapeCheck independently re-fetches the page and finds the claimed value there itself — and the judge's vote is mechanically voided if its restatement of the claim doesn't match what was actually claimed. Absence cannot pass, and substitution cannot pass. Anything unconfirmed returns unverifiable — never pass.

Corroborated by benchmark (full verification only): 0 false passes across 67 frozen labeled cases (26 held out from all calibration, including adversarial traps where the claimed value appears on the page as the wrong thing), and 0 false passes across 21 live-web cases including 9 adversarial traps. Small-N corroboration of the structural guards — not a population accuracy claim.

Two tiers, honestly separated

	Full verification	Presence check
Price	$0.01/row	$0.002/row
Re-fetches the live page	✔	✔
Value present on page	✔	✔
Served live (not stale cache)	✔	✔
Value answers the question (LLM judge + substitution guard)	✔	✘
Positive verdict	pass	present — never pass
check_type	web_field_v1	web_field_presence_v1
Trust score computed	✔	✘

Presence mode confirms the value appears on the live page; it does not confirm the value is the right answer to the question. A presence result is type-incapable of impersonating a full verification, and the published accuracy numbers above apply to full verification only.

How to use it

Chained (recommended): QA every scraper run automatically. Add Scrape QA as an integration on any scraper actor — when the scraper succeeds, Scrape QA verifies its default dataset with zero configuration. Field mapping is inferred automatically from the first rows (and always reported in the summary; set mapping to override).

Batch: run with a datasetId (or inline items). Big dataset? It takes an honest uniform random sample up to maxChecks and reports the population size — never silent truncation.

Standby (per-request, for agents): POST {url, claim, asked} and get one signed verdict back — same contract as the underlying API.

Every output row carries the verdict, confidence, reasons, evidence excerpt, a stable verdict_id, the engine version that produced it, and an ed25519 signature you can verify offline against the public key. The run summary states the trust-score formula in words, counts any fields that appeared outside the mapping (unexamined is never presented as verified), and reports early stops honestly (stopped_early, stop_reason) — the actor never does work it cannot charge for.

Scope (v1)

Server-rendered pages. Client-rendered (JS-only) content returns unverifiable — never a false fail, never a false pass. One set of claimed fields per row (e.g. price + stock on a product page). No headless browser — which also keeps compute costs near zero.

Why trust this actor's own claims?

Because you don't have to. Verdicts are signed; the signing key's public half is served at /pubkey; every verdict carries an engine digest identifying exactly which prompt+calibration produced it; and the fail-safe is mechanical: anything unconfirmed — a skipped check, a failed fetch, a judge that errors — returns unverifiable, never pass.