Pricing

from $0.50 / 1,000 chunk auditeds

RAG Dataset Linter

Audit RAG chunks for duplicates, broken ordering, excessive overlap, missing provenance, and malformed content before vector database ingestion.

Pricing

from $0.50 / 1,000 chunk auditeds

Rating

0.0

(0)

Developer

Sebastián S

Actor stats

Bookmarked

Total users

Monthly active users

17 days ago

Last modified

Categories

Developer tools

You can access the RAG Dataset Linter programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

# Set API token
$API_TOKEN=<YOUR_API_TOKEN>

# Prepare Actor input
$cat > input.json << 'EOF'
<{
<  "records": [
<    {
<      "chunkId": "guide-0",
<      "documentId": "guide",
<      "sourceUrl": "https://example.com/guide",
<      "chunkIndex": 0,
<      "title": "Getting started",
<      "headingPath": [
<        "Getting started"
<      ],
<      "chunkText": "Install the package, create a client, and verify the connection before indexing production content."
<    },
<    {
<      "chunkId": "guide-1",
<      "documentId": "guide",
<      "sourceUrl": "https://example.com/guide",
<      "chunkIndex": 1,
<      "title": "Getting started",
<      "headingPath": [
<        "Getting started"
<      ],
<      "chunkText": "Install the package, create a client, and verify the connection before indexing production content."
<    },
<    {
<      "chunkId": "guide-3",
<      "documentId": "guide",
<      "sourceUrl": "https://example.com/guide",
<      "chunkIndex": 3,
<      "title": "Configuration",
<      "headingPath": [
<        "Configuration"
<      ],
<      "chunkText": "Configure the index name, embedding dimensions, and metadata fields used for retrieval."
<    },
<    {
<      "chunkId": "overlap-0",
<      "documentId": "overlap",
<      "sourceUrl": "https://example.com/overlap",
<      "chunkIndex": 0,
<      "title": "Overlap example",
<      "headingPath": [
<        "Overlap example"
<      ],
<      "chunkText": "This intentionally repeated sample demonstrates an oversized chunk. This intentionally repeated sample demonstrates an oversized chunk. This intentionally repeated sample demonstrates an oversized chunk. This intentionally repeated sample demonstrates an oversized chunk. Verify the source records before embedding them. Keep stable identifiers and source URLs."
<    },
<    {
<      "chunkId": "missing-provenance",
<      "chunkIndex": 0,
<      "chunkText": "As shown above, use the same configuration for the next operation."
<    },
<    {
<      "chunkId": "overlap-1",
<      "documentId": "overlap",
<      "sourceUrl": "https://example.com/overlap",
<      "chunkIndex": 1,
<      "chunkText": "Verify the source records before embedding them. Keep stable identifiers and source URLs. Keep stable identifiers and source URLs for every chunk."
<    }
<  ]
<}
<EOF

# Run the Actor using an HTTP API
# See the full API reference at https://docs.apify.com/api/v2
$curl "https://api.apify.com/v2/acts/sebastian-actors~rag-dataset-linter/runs?token=$API_TOKEN" \
<  -X POST \
<  -d @input.json \
<  -H 'Content-Type: application/json'

RAG Dataset Linter API

Below, you can find a list of relevant HTTP API endpoints for calling the RAG Dataset Linter Actor. For this, you’ll need an Apify account. Replace <YOUR_API_TOKEN> in the URLs with your Apify API token, which you can find under Integrations in Apify Console. For details, see the API reference.

Run Actor

POST
https://api.apify.com/v2/acts/sebastian-actors~rag-dataset-linter/runs?token=<YOUR_API_TOKEN>

Note: By adding the method=POST query parameter, this API endpoint can be called using a GET request and thus used in third-party webhooks. Please refer to our Run Actor API documentation.

Run Actor synchronously and get dataset items

POST
https://api.apify.com/v2/acts/sebastian-actors~rag-dataset-linter/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>

Note: This endpoint supports both POST and GET request methods. However, only the POST method allows you to pass input data. For more information, please refer to our Run Actor synchronously and get dataset items API documentation.

Get Actor

GET
https://api.apify.com/v2/acts/sebastian-actors~rag-dataset-linter?token=<YOUR_API_TOKEN>

For more information, please refer to our Get Actor API documentation.

Actors can be used to scrape web pages, extract data, or automate browser tasks. Use the RAG Dataset Linter API programmatically via the Apify API.

You can choose from:

RAG Dataset Linter API in Python

RAG Dataset Linter API in JavaScript

RAG Dataset Linter API through CLI

RAG Dataset Linter OpenAPI definition

You can start RAG Dataset Linter with the Apify API by sending an HTTP POST request to the Run Actorendpoint. An Actor’s input and its content type can be passed as a payload of the POST request, and additional options can be specified using URL query parameters. The RAG Dataset Linter is identified within the API by its ID, which is the creator’s username and the name of the Actor.

When the RAG Dataset Linter run finishes you can list the data from its default dataset(storage) via the API or you can preview the data directly on Apify Console.

Docs To Rag

gabrielaxy/docs-to-rag

Transform documentation websites into RAG-ready chunks with semantic understanding, quality scoring, and direct vector database integration.

Gabriel Antony Xaviour

RAG Web Browser

travelmonitorlab/rag-web-browser

Search the web and extract content for AI/RAG pipelines. Returns clean text ready for LLM ingestion.

Travel Monitor Lab

Website to RAG Dataset

sebastian-actors/website-to-rag-dataset

Convert public websites, docs, blogs, and XML sitemaps into clean Markdown, structured metadata, and stable chunks for RAG pipelines and vector databases.

Sebastián S

MCP Website RAG Ingestion Tool

glowing_glove/mcp-website-rag-ingestion-tool

MCP-ready website ingestion for AI agents using a prompt-first interface that turns public webpages into source-linked RAG chunks with markdown and token estimates.

Ushba Khan

PDF to RAG Markdown Chunks for Embeddings

awesome_highboy/docforge

Convert PDFs into token-bounded Markdown chunks for RAG, embeddings, and vector databases (Pinecone, Chroma, Weaviate, Qdrant). Set maxTokens + overlap; get clean chunks with page number, token count, and SHA-256 content hash for dedup. JSON dataset ready for any LLM pipeline.

Adam

YouTube Transcript to RAG Dataset

ghostgrid/youtube-transcript-to-rag

Extract YouTube transcripts and convert them into RAG-ready chunks with timestamps, metadata, and content hashes.

GhostGrid

Text Splitter & Chunker for RAG / LLMs

zenomastro/text-splitter-for-llm

Split text into clean, overlapping chunks ready for embeddings, vector databases, RAG and LLM context. Configurable size, overlap, and split strategy.

Rosario Vitale

Rag Embedding Generator

labrat011/rag-embedding-generator

Generate vector embeddings from text or chunked datasets using OpenAI or Cohere. Chains with RAG Content Chunker for end-to-end RAG pipelines. Outputs raw vectors ready for any vector database.

mick_

AI Content Crawler

kai-agent/ai-content-crawler

Crawl any website and get clean, AI-ready content in markdown format. Perfect for RAG pipelines, LLM training data, and vector database ingestion. Features smart chunking, metadata extraction, and multiple output formats.

Kai Agent

Wikipedia to RAG — Article Scraper for AI Pipelines

yuchiaoniu/wikipedia-rag-scraper

Search Wikipedia and download articles as clean Markdown chunks ready for RAG pipelines, Pinecone, Weaviate, Chroma, or any vector database. No API key required.