RAG Evidence & Citation Auditor

Deterministic release checks for RAG answers. The auditor measures citation coverage, validates cited source IDs, checks lexical and numeric support, verifies source hashes, flags stale evidence, and scores provenance completeness.

It does not call an LLM, invent a correctness verdict, or send source data to a model. Reports contain hashes and metrics by default; raw claim text and raw numeric claim values are opt-in.

Input contract

Use a JSON array or NDJSON file. Every record needs:

• question: the user question.
• answer: the generated answer. Inline citations use [source-id].
• sources: evidence objects with id, text, url, retrievedAt, and sha256.
• Optional claims: explicit { "text": "...", "sourceIds": ["..."] } objects for pipelines that already segment claims.

Local CLI

pip install .
rag-evidence-audit examples/good-rag-bundle.json \
  --output reports/report.json \
  --markdown reports/report.md \
  --sarif reports/report.sarif

Exit codes: 0 passed, 1 quality gate failed, 2 input/runtime error.

Apify Actor

The Actor accepts up to five inline or public-HTTPS evaluation bundles. Every successful dataset audit produces one compact summary result and one rag-evidence-audit pay-per-event charge. The detailed JSON and Markdown reports are then stored in the run key-value store under the keys shown in that summary. A detail-storage failure is logged but does not remove the already delivered summary. Invalid inputs produce an uncharged error record.

Suggested initial price: $0.05 per successfully audited dataset. Remove the synthetic apify-default-dataset-item pricing event before enabling the custom event so a dataset write is not charged twice.

What the score means

The score combines citation coverage (30%), deterministic evidence support (35%), provenance completeness (20%), freshness (10%), and citation-reference accuracy (5%). A hash mismatch caps the score below 50. This is an engineering signal, not a claim that an answer is factually correct.

Limits

• Public remote input is HTTPS-only, port 443, with DNS/IP validation and redirect revalidation.
• Remote payloads: 10 MB. Inline Actor input: 2 MB and 500 records. File input: 5,000 records.
• Raw claims, raw numeric claim values, and evidence text are not copied into the report unless includeClaimText is enabled. Numeric values are represented by SHA-256 hashes by default.
• Each record supports at most 100 evidence sources and 200 claims. Source token sets are computed once before bounded near-duplicate comparisons.
• Charged dataset items are compact summaries; detailed reports use key-value-store records so allowed inputs cannot exceed Apify's per-dataset-item size limit.
• Deterministic lexical support can produce false positives or false negatives. Use it as a repeatable release gate, not as a substitute for domain review.

See docs/PRIVACY_AND_LIMITS.md and docs/AUTOMATION_FLOW.md for the operating model.