# RAG Dataset Quality Auditor (`gifted_wagon/rag-dataset-quality-auditor`) Actor

Find duplicate, stale, incomplete, repetitive, and poorly chunked documents before they weaken RAG retrieval.

- **URL**: https://apify.com/gifted\_wagon/rag-dataset-quality-auditor.md
- **Developed by:** [Michael Olmos](https://apify.com/gifted_wagon) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 document auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## RAG Dataset Quality Auditor

Find the documents that make retrieval-augmented generation unreliable **before** they enter your vector database.

RAG Dataset Quality Auditor checks every document or chunk for structural quality problems, returns a filter-ready score, and explains exactly what to fix. It is deterministic, requires no external model key, and keeps document bodies out of its output by default.

### Why use it?

Poor source data creates poor retrieval. Duplicate chunks crowd out better results, oversized documents dilute relevance, stale content produces obsolete answers, and missing source metadata makes answers difficult to verify.

Use this Actor as a quality gate before embedding, after a crawler or migration, or as a scheduled audit for a knowledge base.

### What it detects

| Check | Issue code | Why it matters |
|---|---|---|
| Empty content | `EMPTY_CONTENT` | Nothing useful can be embedded or retrieved |
| Undersized chunks | `TOO_SHORT` | Fragments often lack enough context to answer a query |
| Oversized chunks | `TOO_LONG` | Large records can dilute retrieval precision |
| Missing title or source URL | `MISSING_TITLE`, `MISSING_URL` | Weakens ranking, traceability, and citations |
| Invalid source URL | `INVALID_URL` | Breaks attribution and refresh workflows |
| Stale or invalid timestamps | `STALE_CONTENT`, `INVALID_UPDATED_AT` | Lets obsolete material remain in the index |
| Repetitive content | `REPETITIVE_CONTENT`, `LOW_LEXICAL_DIVERSITY` | Wastes tokens and pollutes search results |
| Exact duplicates | `EXACT_DUPLICATE` | Competing copies can dominate retrieval |
| Near duplicates | `NEAR_DUPLICATE` | Slightly changed copies fragment authority |

Every finding includes its severity, deterministic evidence, and a specific remediation. Each document also receives a quality score from 0 to 100 and an A-F grade.

### Quick start

Run the Actor with no input to audit a safe built-in example, or paste a few documents:

```json
{
  "documents": [
    {
      "id": "kb-101",
      "title": "Reset a customer password",
      "url": "https://example.com/help/reset-password",
      "updatedAt": "2026-08-01T00:00:00.000Z",
      "text": "Paste the complete document or chunk text here. For the default policy, useful chunks contain at least 80 words and no more than 2,000 words."
    }
  ]
}
```

For a production audit, select an existing Apify dataset in the input form:

```json
{
  "datasetId": "YOUR_DATASET_ID",
  "maxDocuments": 500,
  "minWords": 80,
  "maxWords": 2000,
  "staleAfterDays": 365,
  "nearDuplicateThreshold": 0.92,
  "includeContentPreview": false
}
```

The Actor reads the first matching text field from `text`, `markdown`, `content`, or `body`. The field mapping is configurable for custom datasets. Inline documents and a selected dataset can be combined in one run.

### Use it after another Actor

RAG Dataset Quality Auditor accepts the default dataset passed through an Apify integration. Add it from the source Actor's **Integrations** tab and it will audit that run's output automatically. An explicitly selected `datasetId` takes priority when both are present.

Common workflows include:

- crawler output -> quality audit -> vector database
- CMS export -> quality audit -> repair queue
- documentation migration -> duplicate report -> ingestion gate
- scheduled knowledge-base crawl -> freshness audit -> alerting workflow

### Output

The default dataset contains one audit row per charged document. Useful fields include:

- `qualityScore`, `grade`, and severity counts
- `issueCodes` and full `issues` with evidence and remediation
- `wordCount`, `uniqueWordRatio`, and `repetitionRatio`
- `contentHash`, `exactDuplicateOf`, `nearDuplicateOf`, and `similarity`
- `documentId`, `title`, `url`, and `updatedAt`

Example result:

```json
{
  "documentId": "kb-184",
  "qualityScore": 70,
  "grade": "C",
  "issueCodes": ["STALE_CONTENT", "NEAR_DUPLICATE"],
  "nearDuplicateOf": "kb-031",
  "similarity": 0.9481,
  "issues": [
    {
      "code": "STALE_CONTENT",
      "severity": "medium",
      "category": "freshness",
      "message": "The document exceeds the configured freshness window.",
      "evidence": "512 days old; the configured maximum is 365 days.",
      "recommendation": "Review the source, refresh the content, and record a new verified update timestamp."
    }
  ]
}
```

The `OUTPUT` record in the default key-value store contains run totals, grade distribution, issue counts, average quality score, applied thresholds, source type, pricing mode, and whether the run stopped at its spending limit.

### Automate through the API

Start a run with the Apify API:

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/gifted_wagon~rag-dataset-quality-auditor/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"datasetId":"YOUR_DATASET_ID","maxDocuments":500}'
```

The run response contains links and storage IDs for retrieving the document results and summary. You can also call the Actor from the Apify JavaScript or Python client, schedules, webhooks, Make, Zapier, or another Actor.

### Privacy and security

- The Actor requests limited permissions and reads only the dataset selected by the user or supplied by an integration.
- It does not crawl source URLs or send document content to an external AI model.
- Document bodies are not copied to results unless `includeContentPreview` is enabled.
- Content hashes are calculated after Unicode and whitespace normalization.
- Input and output storage follow the access settings of the account running the Actor.

### Pricing

You pay only for completed `document-audited` events. The launch tiers are:

| Apify tier | Price per 1,000 audited documents |
|---|---:|
| Free | $1.50 |
| Bronze | $1.00 |
| Silver | $0.75 |
| Gold | $0.50 |

There is also Apify's standard $0.00005 Actor-start event. The Actor checks the run spending limit before doing paid work and stops cleanly when the remaining budget cannot cover another document.

### Verified behavior

The public-beta benchmark audited 35 documents and 51,612 words across clean documentation chunks, oversized raw manuals, and a controlled noisy migration. It detected all 10 planted defects, classified all seven oversized manuals, produced identical local and cloud results, and completed each cloud profile in under four seconds. Content previews remained disabled throughout the benchmark.

### Scope and limitations

This Actor identifies deterministic structural risks. It does **not** verify factual correctness, source authority, answer relevance, embedding quality, or whether a document contains every fact your users need. Use its output as one quality gate in a broader RAG evaluation process.

Near-duplicate detection uses five-word shingles, a bounded candidate index, and full Jaccard verification. Results are deterministic for the same content, ordering, thresholds, and run date.

### Support

If a result looks wrong or your dataset uses an unsupported shape, open an issue from this Actor's **Issues** tab. Include the run ID, affected issue code, field mapping, and a redacted example when possible. Never post private document content in a public issue.

# Actor input Schema

## `datasetId` (type: `string`):

Select an existing Apify dataset. Each item should contain a text-like field such as text, markdown, content, or body.

## `documents` (type: `array`):

Optional documents pasted directly. Each item may contain id, url, title, text, updatedAt, and metadata.

## `maxDocuments` (type: `integer`):

Maximum number of documents to read and charge for across both sources.

## `minWords` (type: `integer`):

Documents below this threshold receive a too-short issue.

## `maxWords` (type: `integer`):

Documents above this threshold receive a too-long issue and should usually be chunked.

## `staleAfterDays` (type: `integer`):

Documents with a valid updatedAt older than this threshold receive a stale-content issue. Set 0 to disable.

## `nearDuplicateThreshold` (type: `number`):

Jaccard similarity threshold for candidate documents identified by the fingerprint index.

## `contentFields` (type: `array`):

First matching field is used as document text when reading a dataset.

## `titleFields` (type: `array`):

First matching field is used as the document title when reading a dataset.

## `urlFields` (type: `array`):

First matching field is used as the canonical source URL when reading a dataset.

## `updatedAtFields` (type: `array`):

First matching field is used as the freshness timestamp when reading a dataset.

## `includeContentPreview` (type: `boolean`):

Include a short normalized preview in each result. Leave off for sensitive corpora.

## `contentPreviewCharacters` (type: `integer`):

Maximum number of normalized document characters to include when previews are enabled.

## `payload` (type: `object`):

Automatically supplied by Apify when this Actor is connected to another Actor run.

## Actor input object example

```json
{
  "documents": [],
  "maxDocuments": 500,
  "minWords": 80,
  "maxWords": 2000,
  "staleAfterDays": 365,
  "nearDuplicateThreshold": 0.92,
  "contentFields": [
    "text",
    "markdown",
    "content",
    "body"
  ],
  "titleFields": [
    "title",
    "name",
    "heading"
  ],
  "urlFields": [
    "url",
    "sourceUrl",
    "canonicalUrl"
  ],
  "updatedAtFields": [
    "updatedAt",
    "lastModified",
    "modifiedAt",
    "publishedAt"
  ],
  "includeContentPreview": false,
  "contentPreviewCharacters": 280
}
```

# Actor output Schema

## `results` (type: `string`):

One explainable quality-audit result per processed document.

## `summary` (type: `string`):

Run-level totals, grade distribution, issue counts, thresholds, and budget status.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "documents": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("gifted_wagon/rag-dataset-quality-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "documents": [] }

# Run the Actor and wait for it to finish
run = client.actor("gifted_wagon/rag-dataset-quality-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "documents": []
}' |
apify call gifted_wagon/rag-dataset-quality-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,gifted_wagon/rag-dataset-quality-auditor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/a3APhXMuecsBLmKPe/builds/gwd95YbEO6VY3Rlkh/openapi.json