rag-regression-evaluator
Pricing
Pay per usage
Compare baseline and candidate RAG outputs using lexical metrics. Get per-case score changes, regression flags, and a pass/fail threshold decision for up to 100 test cases. No LLM calls. Experimental: does not verify factual accuracy or semantic correctness.
1–100 cases. Unique ID (128 chars), query (4000 chars), baseline/candidate with answer and 0–20 contexts. Each answer/reference/context at most 20000 chars. See README for full contract.
[ { "id": "capital-france", "query": "What is the capital of France?", "reference_answer": "Paris", "baseline": { "contexts": [ "France is a country in Western Europe. Its capital is Paris." ], "answer": "The capital of France is Paris." }, "candidate": { "contexts": [ "France is a country in Western Europe. Its capital is Paris." ], "answer": "The capital of France is Berlin." } }, { "id": "largest-planet", "query": "What is the largest planet in the Solar System?", "reference_answer": "Jupiter", "baseline": { "contexts": [ "Jupiter is the largest planet in the Solar System." ], "answer": "Jupiter is the largest planet in the Solar System." }, "candidate": { "contexts": [ "Jupiter is the largest planet in the Solar System." ], "answer": "Jupiter is the largest planet in the Solar System." } }]