RAG Dataset Quality Auditor

Find the documents that make retrieval-augmented generation unreliable before they enter your vector database.

RAG Dataset Quality Auditor checks every document or chunk for structural quality problems, returns a filter-ready score, and explains exactly what to fix. It is deterministic, requires no external model key, and keeps document bodies out of its output by default.

Why use it?

Poor source data creates poor retrieval. Duplicate chunks crowd out better results, oversized documents dilute relevance, stale content produces obsolete answers, and missing source metadata makes answers difficult to verify.

Use this Actor as a quality gate before embedding, after a crawler or migration, or as a scheduled audit for a knowledge base.

What it detects

Check	Issue code	Why it matters
Empty content	EMPTY_CONTENT	Nothing useful can be embedded or retrieved
Undersized chunks	TOO_SHORT	Fragments often lack enough context to answer a query
Oversized chunks	TOO_LONG	Large records can dilute retrieval precision
Missing title or source URL	MISSING_TITLE, MISSING_URL	Weakens ranking, traceability, and citations
Invalid source URL	INVALID_URL	Breaks attribution and refresh workflows
Stale or invalid timestamps	STALE_CONTENT, INVALID_UPDATED_AT	Lets obsolete material remain in the index
Repetitive content	REPETITIVE_CONTENT, LOW_LEXICAL_DIVERSITY	Wastes tokens and pollutes search results
Exact duplicates	EXACT_DUPLICATE	Competing copies can dominate retrieval
Near duplicates	NEAR_DUPLICATE	Slightly changed copies fragment authority

Every finding includes its severity, deterministic evidence, and a specific remediation. Each document also receives a quality score from 0 to 100 and an A-F grade.

Quick start

Run the Actor with no input to audit a safe built-in example, or paste a few documents:

{
  "documents": [
    {
      "id": "kb-101",
      "title": "Reset a customer password",
      "url": "https://example.com/help/reset-password",
      "updatedAt": "2026-08-01T00:00:00.000Z",
      "text": "Paste the complete document or chunk text here. For the default policy, useful chunks contain at least 80 words and no more than 2,000 words."
    }
  ]
}

For a production audit, select an existing Apify dataset in the input form:

{
  "datasetId": "YOUR_DATASET_ID",
  "maxDocuments": 500,
  "minWords": 80,
  "maxWords": 2000,
  "staleAfterDays": 365,
  "nearDuplicateThreshold": 0.92,
  "includeContentPreview": false
}

The Actor reads the first matching text field from text, markdown, content, or body. The field mapping is configurable for custom datasets. Inline documents and a selected dataset can be combined in one run.

Use it after another Actor

RAG Dataset Quality Auditor accepts the default dataset passed through an Apify integration. Add it from the source Actor's Integrations tab and it will audit that run's output automatically. An explicitly selected datasetId takes priority when both are present.

Common workflows include:

• crawler output -> quality audit -> vector database
• CMS export -> quality audit -> repair queue
• documentation migration -> duplicate report -> ingestion gate
• scheduled knowledge-base crawl -> freshness audit -> alerting workflow

Output

The default dataset contains one audit row per charged document. Useful fields include:

• qualityScore, grade, and severity counts
• issueCodes and full issues with evidence and remediation
• wordCount, uniqueWordRatio, and repetitionRatio
• contentHash, exactDuplicateOf, nearDuplicateOf, and similarity
• documentId, title, url, and updatedAt

Example result:

{
  "documentId": "kb-184",
  "qualityScore": 70,
  "grade": "C",
  "issueCodes": ["STALE_CONTENT", "NEAR_DUPLICATE"],
  "nearDuplicateOf": "kb-031",
  "similarity": 0.9481,
  "issues": [
    {
      "code": "STALE_CONTENT",
      "severity": "medium",
      "category": "freshness",
      "message": "The document exceeds the configured freshness window.",
      "evidence": "512 days old; the configured maximum is 365 days.",
      "recommendation": "Review the source, refresh the content, and record a new verified update timestamp."
    }
  ]
}

The OUTPUT record in the default key-value store contains run totals, grade distribution, issue counts, average quality score, applied thresholds, source type, pricing mode, and whether the run stopped at its spending limit.

Automate through the API

Start a run with the Apify API:

curl -X POST \
  "https://api.apify.com/v2/acts/gifted_wagon~rag-dataset-quality-auditor/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"datasetId":"YOUR_DATASET_ID","maxDocuments":500}'

The run response contains links and storage IDs for retrieving the document results and summary. You can also call the Actor from the Apify JavaScript or Python client, schedules, webhooks, Make, Zapier, or another Actor.

Privacy and security

• The Actor requests limited permissions and reads only the dataset selected by the user or supplied by an integration.
• It does not crawl source URLs or send document content to an external AI model.
• Document bodies are not copied to results unless includeContentPreview is enabled.
• Content hashes are calculated after Unicode and whitespace normalization.
• Input and output storage follow the access settings of the account running the Actor.

Pricing

You pay only for completed document-audited events. The launch tiers are:

Apify tier	Price per 1,000 audited documents
Free	$1.50
Bronze	$1.00
Silver	$0.75
Gold	$0.50

There is also Apify's standard $0.00005 Actor-start event. The Actor checks the run spending limit before doing paid work and stops cleanly when the remaining budget cannot cover another document.

Verified behavior

The public-beta benchmark audited 35 documents and 51,612 words across clean documentation chunks, oversized raw manuals, and a controlled noisy migration. It detected all 10 planted defects, classified all seven oversized manuals, produced identical local and cloud results, and completed each cloud profile in under four seconds. Content previews remained disabled throughout the benchmark.

Scope and limitations

This Actor identifies deterministic structural risks. It does not verify factual correctness, source authority, answer relevance, embedding quality, or whether a document contains every fact your users need. Use its output as one quality gate in a broader RAG evaluation process.

Near-duplicate detection uses five-word shingles, a bounded candidate index, and full Jaccard verification. Results are deterministic for the same content, ordering, thresholds, and run date.

Support

If a result looks wrong or your dataset uses an unsupported shape, open an issue from this Actor's Issues tab. Include the run ID, affected issue code, field mapping, and a redacted example when possible. Never post private document content in a public issue.