# Detect duplicate embedding spend

**Use case:** 

Find exact duplicate RAG chunks and excessive overlap, estimate wasted embedding tokens, and stop redundant vectors before indexing.

## Input

```json
{
  "records": [
    {
      "chunkId": "pricing-0",
      "documentId": "pricing",
      "sourceUrl": "https://example.com/pricing",
      "chunkIndex": 0,
      "title": "Pricing",
      "headingPath": [
        "Embedding costs"
      ],
      "chunkText": "Embedding duplicate chunks wastes tokens and creates redundant vectors that can crowd retrieval results."
    },
    {
      "chunkId": "faq-0",
      "documentId": "faq",
      "sourceUrl": "https://example.com/faq",
      "chunkIndex": 0,
      "title": "FAQ",
      "headingPath": [
        "Embedding costs"
      ],
      "chunkText": "Embedding duplicate chunks wastes tokens and creates redundant vectors that can crowd retrieval results."
    },
    {
      "chunkId": "operations-0",
      "documentId": "operations",
      "sourceUrl": "https://example.com/operations",
      "chunkIndex": 0,
      "title": "Operations",
      "headingPath": [
        "Cost control"
      ],
      "chunkText": "Audit each new corpus before embedding so repeated content and excessive adjacent overlap are visible as estimated wasted tokens."
    }
  ],
  "maxItems": 1000,
  "fieldMapping": {},
  "minTokens": 5,
  "maxTokens": 2000,
  "nearDuplicateSimilarity": 0.95,
  "maxAdjacentOverlapRatio": 0.35,
  "enableHeuristicWarnings": true,
  "maxFindings": 5000,
  "includeMarkdownReport": true,
  "failRunOnError": false
}
```

## Output

```json
{
  "gateStatus": {
    "label": "Gate",
    "format": "string"
  },
  "automationAction": {
    "label": "Action",
    "format": "string"
  },
  "sourceType": {
    "label": "Source type",
    "format": "string"
  },
  "sourceId": {
    "label": "Source ID",
    "format": "string"
  },
  "itemsAudited": {
    "label": "Audited",
    "format": "integer"
  },
  "findingCount": {
    "label": "Findings",
    "format": "integer"
  },
  "errorCount": {
    "label": "Errors",
    "format": "integer"
  },
  "warningCount": {
    "label": "Warnings",
    "format": "integer"
  },
  "exactDuplicateRate": {
    "label": "Exact duplicate rate",
    "format": "number"
  },
  "nearDuplicateRate": {
    "label": "Near duplicate rate",
    "format": "number"
  },
  "estimatedEmbeddingWasteTokens": {
    "label": "Estimated wasted tokens",
    "format": "integer"
  },
  "truncated": {
    "label": "Truncated",
    "format": "boolean"
  },
  "finishedAt": {
    "label": "Finished",
    "format": "string"
  }
}
```

## About this Actor

This example demonstrates how to use [RAG Dataset Linter](https://apify.com/sebastian-actors/rag-dataset-linter) with a specific input configuration. Visit the [Actor detail page](https://apify.com/sebastian-actors/rag-dataset-linter) to learn more, explore other use cases, and run it yourself.