# Rag Evaluation Dataset Generator (`systra_ai/systra-ai-rag-evaluation-dataset-generator`) Actor

Generate citation-grounded RAG evaluation datasets from websites, sitemaps, PDFs, text, and inline documents. Create validated factual, multi-hop, unanswerable, and adversarial test cases with exact evidence quotes, grading hints, and JSONL, CSV, or RAGAS-ready exports.

- **URL**: https://apify.com/systra\_ai/systra-ai-rag-evaluation-dataset-generator.md
- **Developed by:** [Troy Hendrick](https://apify.com/systra_ai) (community)
- **Categories:** AI, Automation, Developer tools
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $90.00 / 1,000 schema validateds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## RAG Evaluation Dataset Generator

Generate citation-grounded RAG evaluation datasets from public websites, sitemaps, PDFs, text/JSON files, or inline documents. The Actor produces test questions, reference answers, exact evidence quotes, grading hints, and portable exports.

This is an Apify Actor designed for pay-per-event monetization. Each validated case written to the default dataset maps to one `evaluation-case` event.

### What makes the output trustworthy

The model is not trusted to cite itself correctly. After generation, the Actor independently checks every answerable case:

- Every cited chunk ID must exist in the evidence batch.
- Every evidence quote must appear in that chunk after whitespace normalization.
- Answerable cases must cite evidence.
- Unanswerable cases must not cite evidence.
- Multi-hop cases must cite at least two chunks.
- Near-duplicate questions are rejected.

Candidates that fail these checks are not written to the dataset and are not billed as evaluation cases.

### Supported inputs

- Individual public webpages
- JavaScript-rendered webpages through automatic Chromium fallback
- Same-domain documentation crawling
- XML sitemaps and sitemap indexes
- Public PDF URLs
- Plain-text and JSON URLs
- Inline documents supplied in the Actor input

The fetcher follows at most five redirects, enforces a per-source byte limit, rejects private/local network destinations, and validates every redirect destination to reduce SSRF risk.

### Evaluation case types

| Type | Purpose |
| --- | --- |
| `factual` | Direct retrieval of an explicit fact |
| `paraphrase` | Natural wording different from the source |
| `procedural` | Ordered steps or required actions |
| `policy` | Eligibility, restrictions, exceptions, or obligations |
| `numerical` | Amounts, limits, prices, quantities, or thresholds |
| `temporal` | Deadlines, durations, dates, or sequencing |
| `multi_hop` | Requires evidence from two or more chunks |
| `unanswerable` | Plausible question absent from the collection |
| `adversarial` | False premise or pressure to invent unsupported facts |

### Quick start

Requirements:

- Node.js 22 or later
- An OpenAI API key
- An Apify account and CLI for local Actor execution or deployment

Install and test:

```bash
npm install
npm test
```

For local Apify execution, create `storage/key_value_stores/default/INPUT.json` using one of the files in [`examples`](./examples), then run:

```bash
export OPENAI_API_KEY="your-key"
npx apify run --purge
```

The API key can alternatively be provided using the secret `openAIApiKey` input. For a published Actor, prefer setting `OPENAI_API_KEY` as an Actor environment secret so customers do not need to supply credentials.

### Example input

```json
{
  "sourceUrls": ["https://example.com/help/sitemap.xml"],
  "crawlMode": "sitemap",
  "browserRendering": "auto",
  "maxPages": 30,
  "testCaseCount": 30,
  "questionTypes": [
    "factual",
    "paraphrase",
    "procedural",
    "multi_hop",
    "unanswerable",
    "adversarial"
  ],
  "language": "English",
  "model": "gpt-5.6-luna"
}
```

At least one `sourceUrls` or `inlineDocuments` item is required.

### Dataset item

```json
{
  "id": "eval_...",
  "question": "When can an order be cancelled?",
  "expectedAnswer": "An order can be cancelled before it is dispatched.",
  "answerability": "answerable",
  "questionType": "policy",
  "difficulty": "easy",
  "sourceReferences": [
    {
      "sourceId": "source_...",
      "chunkId": "chunk_...",
      "sourceUrl": "https://example.com/cancellations",
      "title": "Cancellation policy",
      "quote": "Customers may cancel an order until it is dispatched."
    }
  ],
  "rationale": "Tests retrieval of the cancellation cutoff.",
  "grading": {
    "mustContain": ["before dispatch"],
    "mustNotContain": ["after dispatch"],
    "acceptableVariations": ["until the order is dispatched"]
  },
  "metadata": {
    "language": "English",
    "generatorModel": "gpt-5.6-luna",
    "generatedAt": "2026-08-24T12:00:00.000Z",
    "sourceChunkIds": ["chunk_..."]
  }
}
```

### Outputs

The default dataset contains one validated evaluation case per item. The default key-value store also contains:

- `EVALUATION_DATASET.jsonl` — complete canonical records
- `EVALUATION_DATASET.csv` — flat export for spreadsheets and generic pipelines
- `RAGAS_DATASET.jsonl` — `user_input`, `reference`, and `reference_contexts`
- `OUTPUT` — generation manifest, counts, source failures, and export names

### Monetization setup

Pricing events are configured in Apify Console, not committed in `actor.json`.

Recommended initial configuration:

| Event | Suggested starting price | Notes |
| --- | ---: | --- |
| `apify-actor-start` | $0.00005 | Keep Apify's synthetic start event |
| `evaluation-case` | $0.08–$0.12 | Charged only after a validated case is saved |

Start with platform usage passed through while measuring real OpenAI and compute costs. Before Store launch, move to event-only pricing if the margin covers model and platform costs; event-only PPE is necessary for Apify's agentic payment eligibility.

The code respects a run's maximum-charge limit and stops saving additional cases after the limit is reached.

### Architecture

```text
URL/PDF/text input
      │
      ▼
Safe fetch and extraction
      │
      ▼
Deterministic chunking
      │
      ▼
OpenAI Structured Output generation
      │
      ▼
Citation and duplicate validation
      │
      ├── rejected candidates → manifest count
      │
      ▼
Apify dataset + JSONL/CSV/Ragas exports
```

Generation uses the OpenAI Responses API with a Zod schema. Source material is explicitly treated as untrusted evidence so instructions embedded in crawled pages are not followed.

### Development

```bash
npm run build
npm test
```

Tests cover:

- Input normalization and invalid limits
- Chunk construction and metadata preservation
- Exact evidence grounding
- Fabricated quote rejection
- Multi-hop citation requirements
- Unanswerable cases
- Private and special-use network address rejection
- End-to-end pipeline behavior with a deterministic model double

### Current limitations

- Browser rendering is used only for HTML pages and increases startup time and memory usage when triggered.
- DOCX files are not yet parsed.
- Evidence matching allows whitespace normalization but otherwise requires verbatim text.
- The Actor creates datasets; it does not execute the target RAG system or calculate retrieval/answer metrics.
- A live generation run incurs OpenAI API costs and is not part of the offline test suite.

### Responsible use

Only process material you are authorised to access and reuse. Respect website terms, robots policies, copyright, privacy obligations, and applicable data-protection law. Do not use the Actor to extract confidential material, bypass authentication, or process personal data without a lawful basis.

# Actor input Schema

## `sourceUrls` (type: `array`):

Public webpages, sitemap XML files, text/JSON documents, or PDFs to use as ground truth.

## `inlineDocuments` (type: `array`):

Optional documents supplied directly as JSON objects with title, content, and optional sourceUrl.

## `crawlMode` (type: `string`):

Process only supplied pages, follow same-domain links, or expand sitemap URLs.

## `browserRendering` (type: `string`):

Automatically render thin HTML pages in Chromium, never use a browser, or render every HTML page.

## `maxPages` (type: `integer`):

Hard cap on fetched source documents.

## `maxDepth` (type: `integer`):

Used only for same-domain crawling.

## `testCaseCount` (type: `integer`):

The Actor may return fewer cases when generated evidence fails grounding or duplicate checks.

## `questionTypes` (type: `array`):

Mix of evaluation cases to generate.

## `language` (type: `string`):

Language name or BCP-47 code for questions and reference answers.

## `model` (type: `string`):

A model that supports Structured Outputs.

## `openAIApiKey` (type: `string`):

Optional. When omitted, the Actor uses its OPENAI\_API\_KEY environment variable.

## `maxChunkCharacters` (type: `integer`):

Approximate maximum source characters provided per evidence chunk.

## `chunkOverlapCharacters` (type: `integer`):

Context repeated between adjacent chunks.

## `generationBatchSize` (type: `integer`):

Smaller batches improve grounding; larger batches reduce model calls.

## `maxSourceBytes` (type: `integer`):

Safety limit for each downloaded document.

## Actor input object example

```json
{
  "sourceUrls": [],
  "inlineDocuments": [],
  "crawlMode": "single-page",
  "browserRendering": "auto",
  "maxPages": 30,
  "maxDepth": 2,
  "testCaseCount": 30,
  "questionTypes": [
    "factual",
    "paraphrase",
    "procedural",
    "multi_hop",
    "unanswerable",
    "adversarial"
  ],
  "language": "English",
  "model": "gpt-5.6-luna",
  "maxChunkCharacters": 5000,
  "chunkOverlapCharacters": 400,
  "generationBatchSize": 6,
  "maxSourceBytes": 10000000
}
```

# Actor output Schema

## `evaluationCases` (type: `string`):

No description

## `jsonl` (type: `string`):

No description

## `csv` (type: `string`):

No description

## `ragas` (type: `string`):

No description

## `manifest` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("systra_ai/systra-ai-rag-evaluation-dataset-generator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("systra_ai/systra-ai-rag-evaluation-dataset-generator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call systra_ai/systra-ai-rag-evaluation-dataset-generator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,systra_ai/systra-ai-rag-evaluation-dataset-generator"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/u0Prm6CZJ8ZmnfVhr/builds/7fddWaaTbFLHWcBvN/openapi.json