Rag Evaluation Dataset Generator

Generate citation-grounded RAG evaluation datasets from public websites, sitemaps, PDFs, text/JSON files, or inline documents. The Actor produces test questions, reference answers, exact evidence quotes, grading hints, and portable exports.

This is an Apify Actor designed for pay-per-event monetization. Each validated case written to the default dataset maps to one evaluation-case event.

What makes the output trustworthy

The model is not trusted to cite itself correctly. After generation, the Actor independently checks every answerable case:

• Every cited chunk ID must exist in the evidence batch.
• Every evidence quote must appear in that chunk after whitespace normalization.
• Answerable cases must cite evidence.
• Unanswerable cases must not cite evidence.
• Multi-hop cases must cite at least two chunks.
• Near-duplicate questions are rejected.

Candidates that fail these checks are not written to the dataset and are not billed as evaluation cases.

Supported inputs

• Individual public webpages
• JavaScript-rendered webpages through automatic Chromium fallback
• Same-domain documentation crawling
• XML sitemaps and sitemap indexes
• Public PDF URLs
• Plain-text and JSON URLs
• Inline documents supplied in the Actor input

The fetcher follows at most five redirects, enforces a per-source byte limit, rejects private/local network destinations, and validates every redirect destination to reduce SSRF risk.

Evaluation case types

Type	Purpose
factual	Direct retrieval of an explicit fact
paraphrase	Natural wording different from the source
procedural	Ordered steps or required actions
policy	Eligibility, restrictions, exceptions, or obligations
numerical	Amounts, limits, prices, quantities, or thresholds
temporal	Deadlines, durations, dates, or sequencing
multi_hop	Requires evidence from two or more chunks
unanswerable	Plausible question absent from the collection
adversarial	False premise or pressure to invent unsupported facts

Quick start

Requirements:

• Node.js 22 or later
• An OpenAI API key
• An Apify account and CLI for local Actor execution or deployment

Install and test:

npm install
npm test

For local Apify execution, create storage/key_value_stores/default/INPUT.json using one of the files in ./examples, then run:

export OPENAI_API_KEY="your-key"
npx apify run --purge

The API key can alternatively be provided using the secret openAIApiKey input. For a published Actor, prefer setting OPENAI_API_KEY as an Actor environment secret so customers do not need to supply credentials.

Example input

{
  "sourceUrls": ["https://example.com/help/sitemap.xml"],
  "crawlMode": "sitemap",
  "browserRendering": "auto",
  "maxPages": 30,
  "testCaseCount": 30,
  "questionTypes": [
    "factual",
    "paraphrase",
    "procedural",
    "multi_hop",
    "unanswerable",
    "adversarial"
  ],
  "language": "English",
  "model": "gpt-5.6-luna"
}

At least one sourceUrls or inlineDocuments item is required.

Dataset item

{
  "id": "eval_...",
  "question": "When can an order be cancelled?",
  "expectedAnswer": "An order can be cancelled before it is dispatched.",
  "answerability": "answerable",
  "questionType": "policy",
  "difficulty": "easy",
  "sourceReferences": [
    {
      "sourceId": "source_...",
      "chunkId": "chunk_...",
      "sourceUrl": "https://example.com/cancellations",
      "title": "Cancellation policy",
      "quote": "Customers may cancel an order until it is dispatched."
    }
  ],
  "rationale": "Tests retrieval of the cancellation cutoff.",
  "grading": {
    "mustContain": ["before dispatch"],
    "mustNotContain": ["after dispatch"],
    "acceptableVariations": ["until the order is dispatched"]
  },
  "metadata": {
    "language": "English",
    "generatorModel": "gpt-5.6-luna",
    "generatedAt": "2026-08-24T12:00:00.000Z",
    "sourceChunkIds": ["chunk_..."]
  }
}

Outputs

The default dataset contains one validated evaluation case per item. The default key-value store also contains:

• EVALUATION_DATASET.jsonl — complete canonical records
• EVALUATION_DATASET.csv — flat export for spreadsheets and generic pipelines
• RAGAS_DATASET.jsonl — user_input, reference, and reference_contexts
• OUTPUT — generation manifest, counts, source failures, and export names

Monetization setup

Pricing events are configured in Apify Console, not committed in actor.json.

Recommended initial configuration:

Event	Suggested starting price	Notes
apify-actor-start	$0.00005	Keep Apify's synthetic start event
evaluation-case	$0.08–$0.12	Charged only after a validated case is saved

Start with platform usage passed through while measuring real OpenAI and compute costs. Before Store launch, move to event-only pricing if the margin covers model and platform costs; event-only PPE is necessary for Apify's agentic payment eligibility.

The code respects a run's maximum-charge limit and stops saving additional cases after the limit is reached.

Architecture

URL/PDF/text input
      │
      ▼
Safe fetch and extraction
      │
      ▼
Deterministic chunking
      │
      ▼
OpenAI Structured Output generation
      │
      ▼
Citation and duplicate validation
      │
      ├── rejected candidates → manifest count
      │
      ▼
Apify dataset + JSONL/CSV/Ragas exports

Generation uses the OpenAI Responses API with a Zod schema. Source material is explicitly treated as untrusted evidence so instructions embedded in crawled pages are not followed.

Development

npm run build
npm test

Tests cover:

• Input normalization and invalid limits
• Chunk construction and metadata preservation
• Exact evidence grounding
• Fabricated quote rejection
• Multi-hop citation requirements
• Unanswerable cases
• Private and special-use network address rejection
• End-to-end pipeline behavior with a deterministic model double

Current limitations

• Browser rendering is used only for HTML pages and increases startup time and memory usage when triggered.
• DOCX files are not yet parsed.
• Evidence matching allows whitespace normalization but otherwise requires verbatim text.
• The Actor creates datasets; it does not execute the target RAG system or calculate retrieval/answer metrics.
• A live generation run incurs OpenAI API costs and is not part of the offline test suite.

Responsible use

Only process material you are authorised to access and reuse. Respect website terms, robots policies, copyright, privacy obligations, and applicable data-protection law. Do not use the Actor to extract confidential material, bypass authentication, or process personal data without a lawful basis.