# Audit Website to RAG Dataset output

**Use case:** 

Check Website to RAG Dataset chunks for duplicates, overlap, sequence gaps, and missing provenance before embedding or indexing them.

## Input

```json
{
  "records": [
    {
      "chunkId": "docs-0",
      "documentId": "https://example.com/docs",
      "url": "https://example.com/docs",
      "chunkIndex": 0,
      "title": "Product documentation",
      "headingPath": [
        "Getting started"
      ],
      "chunkText": "Install the client, configure the API key, and verify connectivity before indexing the website documentation."
    },
    {
      "chunkId": "docs-1",
      "documentId": "https://example.com/docs",
      "url": "https://example.com/docs",
      "chunkIndex": 1,
      "title": "Product documentation",
      "headingPath": [
        "Getting started"
      ],
      "chunkText": "Verify connectivity before indexing the website documentation. Preserve the page URL, heading path, and chunk order for grounded retrieval."
    },
    {
      "chunkId": "support-0",
      "documentId": "https://example.com/support",
      "url": "https://example.com/support",
      "chunkIndex": 0,
      "title": "Support",
      "headingPath": [
        "Troubleshooting"
      ],
      "chunkText": "Use the generated report to find duplicate pages, repeated navigation text, missing provenance, and excessive overlap before embedding the crawl output."
    }
  ],
  "maxItems": 1000,
  "fieldMapping": {
    "text": "chunkText",
    "chunkId": "chunkId",
    "documentId": "documentId",
    "sourceUrl": "url",
    "title": "title",
    "headingPath": "headingPath",
    "chunkIndex": "chunkIndex"
  },
  "minTokens": 10,
  "maxTokens": 2000,
  "nearDuplicateSimilarity": 0.95,
  "maxAdjacentOverlapRatio": 0.35,
  "enableHeuristicWarnings": true,
  "maxFindings": 5000,
  "includeMarkdownReport": true,
  "failRunOnError": false
}
```

## Output

```json
{
  "gateStatus": {
    "label": "Gate",
    "format": "string"
  },
  "automationAction": {
    "label": "Action",
    "format": "string"
  },
  "sourceType": {
    "label": "Source type",
    "format": "string"
  },
  "sourceId": {
    "label": "Source ID",
    "format": "string"
  },
  "itemsAudited": {
    "label": "Audited",
    "format": "integer"
  },
  "findingCount": {
    "label": "Findings",
    "format": "integer"
  },
  "errorCount": {
    "label": "Errors",
    "format": "integer"
  },
  "warningCount": {
    "label": "Warnings",
    "format": "integer"
  },
  "exactDuplicateRate": {
    "label": "Exact duplicate rate",
    "format": "number"
  },
  "nearDuplicateRate": {
    "label": "Near duplicate rate",
    "format": "number"
  },
  "estimatedEmbeddingWasteTokens": {
    "label": "Estimated wasted tokens",
    "format": "integer"
  },
  "truncated": {
    "label": "Truncated",
    "format": "boolean"
  },
  "finishedAt": {
    "label": "Finished",
    "format": "string"
  }
}
```

## About this Actor

This example demonstrates how to use [RAG Dataset Linter](https://apify.com/sebastian-actors/rag-dataset-linter) with a specific input configuration. Visit the [Actor detail page](https://apify.com/sebastian-actors/rag-dataset-linter) to learn more, explore other use cases, and run it yourself.