# RAG Documentation Scraper — LLM-Ready Chunker (`lernowa_ai_labs/rag-docs-scraper`) Actor

Crawl any documentation site and output LLM-ready chunks in LangChain, LlamaIndex, Haystack, or JSONL format. Features heading-boundary chunking, SHA-256 dedup, DynamoDB delta tracking, and AWS Textract for scanned PDFs.

- **URL**: https://apify.com/lernowa_ai_labs/rag-docs-scraper.md
- **Developed by:** [LERNOWA AI LABS PRIVATE LIMITED](https://apify.com/lernowa_ai_labs) (community)
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## RAG Documentation Scraper

**Turn any documentation site into a vector-database-ready knowledge base in one click.**

Crawls documentation sites with Playwright, strips HTML navigation boilerplate, splits content on heading boundaries, and outputs clean Markdown or JSON chunks with embedded source metadata — ready to drop into Pinecone, Chroma, pgvector, or any LangChain / LlamaIndex / Haystack pipeline.

***

### Why AI engineers choose this over generic scrapers

| Generic scraper output | What you get here |
|------------------------|-------------------|
| Raw HTML with nav bars, sidebars, cookie banners | Clean text extracted from the main content area only — no noise |
| One blob per page | Heading-boundary chunks that map naturally to retrieval units |
| No metadata | Every chunk carries `source_url`, `page_title`, `section` breadcrumb, `token_count`, `content_hash` |
| Re-processes every page on every run | Optional DynamoDB delta tracking skips unchanged pages — up to **80% cost reduction** on re-runs |
| Duplicate content from versioned docs | SHA-256 deduplication silently drops identical chunks regardless of URL |

***

### Benchmarks

Tested against the Crawlee.dev documentation (production run, 2026-10-18):

- **25 pages crawled → 94 chunks** produced, **0 errors**
- All chunks under 512 tokens (compatible with text-embedding-3-small, nomic-embed-text, bge-m3)
- Per-page time: ~1.8 s average (Playwright, images/fonts blocked)
- Noise stripped: nav, sidebar, footer, cookie banners, pagination, breadcrumbs, "edit this page" widgets

***

### 3-line Python quickstart

```python
from apify_client import ApifyClient
chunks = ApifyClient("YOUR_APIFY_TOKEN").actor("YOUR_USERNAME/apify-rag-docs-scraper").call(
    run_input={"startUrls": [{"url": "https://YOUR_DOCS_SITE.com/docs/"}], "chunkSize": 500, "outputFormat": "json-chunks"}
)["defaultDatasetId"]
items = list(ApifyClient("YOUR_APIFY_TOKEN").dataset(chunks).iterate_items())
## items[0] → {"text": "...", "metadata": {"source": "...", "title": "...", "section": "...", "token_count": 487}}
```

See `developer_quickstart/colab_rag_example.ipynb` for a full end-to-end walkthrough: actor run → dataset pull → ChromaDB insert → semantic search query.

***

### Key features

- **3 RAG-native output formats** — `markdown` (default), `json-chunks` (Pinecone/Chroma/pgvector ready), `text` (bare text)
- **4 framework-specific formats** — LangChain `Document`, LlamaIndex `TextNode`, Haystack `Document`, raw JSONL
- **Flexible chunk size** — Set `chunkSize` (tokens) to control granularity; works with any embedding model
- **4 chunking strategies** — heading boundary (recommended), fixed token window with overlap, paragraph, whole-page
- **PDF support** — basic pdfjs extraction (free) or AWS Textract OCR for scanned/image PDFs
- **Incremental crawling** — DynamoDB delta tracking; re-runs skip unchanged pages entirely
- **Version-aware URL filtering** — restrict to a specific version path (e.g. `/v2/`) with one input field
- **MCP-compatible** — includes an MCP server wrapper so Claude Desktop and Cursor can call this actor as a live tool

***

### Output formats

#### `markdown` (default) — generic RAG, vector-DB agnostic

```json
{
  "markdown": "## Quick Start\n\nCrawlee is available as a set of NPM packages...",
  "source": "https://crawlee.dev/docs/introduction",
  "title": "Introduction | Crawlee for JavaScript",
  "section": "Introduction > Quick Start",
  "chunk_index": 0,
  "chunk_total": 12,
  "token_count": 312,
  "content_hash": "c4a7e891f23b5d10",
  "scraped_at": "2026-10-18T14:45:22.341Z"
}
```

#### `json-chunks` — Pinecone / Chroma / pgvector

```json
{
  "text": "Quick Start\n\nCrawlee is available as a set of NPM packages...",
  "metadata": {
    "source": "https://crawlee.dev/docs/introduction",
    "title": "Introduction | Crawlee for JavaScript",
    "section": "Introduction > Quick Start",
    "chunk_index": 0,
    "chunk_total": 12,
    "token_count": 312,
    "last_modified": null,
    "content_hash": "c4a7e891f23b5d10",
    "scraped_at": "2026-10-18T14:45:22.341Z"
  }
}
```

#### `langchain` — drop-in LangChain `Document`

```json
{
  "page_content": "Quick Start\n\nCrawlee is available as a set of NPM packages...",
  "metadata": {
    "source": "https://crawlee.dev/docs/introduction",
    "heading_path": "Introduction > Quick Start",
    "chunk_index": 0,
    "chunk_total": 12,
    "token_count": 312,
    "content_hash": "c4a7e891f23b5d10"
  }
}
```

***

### Input parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `startUrls` | array | — | **Required.** Root URL(s) to crawl |
| `urlPatterns` | string\[] | `[]` | Glob patterns to restrict crawl scope (e.g. `/docs/**`). Empty = all same-domain URLs |
| `maxDepth` | integer | `5` | Link levels to follow (0 = start pages only) |
| `outputFormat` | enum | `markdown` | `markdown`, `json-chunks`, `text`, `langchain`, `llamaindex`, `haystack`, `raw_jsonl` |
| `chunkSize` | integer | `512` | Target token count per chunk. 500 works well for most embedding models |
| `chunkStrategy` | enum | `heading_boundary` | `heading_boundary`, `fixed_tokens`, `paragraph`, `page` |
| `chunkOverlapTokens` | integer | `50` | Token overlap between adjacent chunks (for `fixed_tokens` strategy) |
| `deduplicateContent` | boolean | `true` | SHA-256 dedup — skip chunks already seen in this run |
| `versionFilter` | string | `""` | Only crawl URLs containing this path fragment (e.g. `/v3/`) |
| `useDeltaTracking` | boolean | `false` | DynamoDB incremental crawling — re-runs skip unchanged pages |
| `pdfHandling` | enum | `skip` | `skip`, `extract_text`, `aws_textract` |
| `maxRequestsPerCrawl` | integer | `200` | Hard page cap per run. `0` = unlimited |
| `excludeSelectors` | string\[] | `[]` | Extra CSS selectors to strip (e.g. `.announcement-bar`) |

***

### Pricing

This actor uses **Pay-Per-Result** pricing — you pay per chunk pushed to the dataset.

| Docs site size | Typical chunks | Estimated cost |
|----------------|----------------|----------------|
| 50 pages | 250–400 chunks | ~$1.25–$2.00 |
| 100 pages | 500–800 chunks | ~$2.50–$4.00 |
| 500 pages | 2,500–4,000 chunks | ~$12.50–$20.00 |

**Cut costs by up to 80%:** Enable `useDeltaTracking: true` so re-runs only process pages that changed since the last run. Stable documentation sites rarely change more than 10–20% of pages per day.

***

### Pre-configured Published Tasks

Five ready-to-run tasks are available in the Apify Store for the most popular documentation niches:

1. **LangChain Documentation Scraper** — Full LangChain Python docs, Markdown + RAG chunks
2. **Stripe API Docs Ingestion Tool** — Stripe API reference in `json-chunks` format for payment bots
3. **Docusaurus / ReadTheDocs Crawler** — Universal adapter for Docusaurus and ReadTheDocs sites
4. **Cursor / Claude MCP Documentation Extractor** — MCP spec + Cursor AI docs into structured chunks
5. **GitHub Wiki & Markdown Repository Scraper** — GitHub Wikis and Markdown repos

Run `python scripts/create_published_tasks.py` (after `apify push`) to create these tasks programmatically.

***

### AWS Integration (optional)

AWS features — Textract OCR for PDFs and DynamoDB delta tracking — are fully optional. Set these environment variables in Apify Console → Actor Settings → Environment Variables:

| Variable | Required for |
|----------|-------------|
| `AWS_ACCESS_KEY_ID` | Delta tracking + Textract |
| `AWS_SECRET_ACCESS_KEY` | Delta tracking + Textract |
| `AWS_S3_BUCKET` | Textract only (temp staging) |

The DynamoDB table is created automatically on first use (PAY_PER_REQUEST billing, 90-day TTL).

***

### MCP / Agent integration

Include this actor as a live tool in Claude Desktop or Cursor — see `developer_quickstart/mcp_client_config_example.json` for the exact config block. The MCP server exposes two tools:

- **`scrape_and_chunk`** — crawl a URL and return RAG-ready chunks
- **`get_actor_status`** — poll a running actor run by ID

***

### FAQ

**The actor crawls only one page — why aren't more pages enqueued?**

A redirect often changes the URL structure after navigation. Check the actual URL in the logs (it may be `/en/docs/` instead of `/docs/`) and update `startUrls` and `urlPatterns` to match the post-redirect path.

**Chunks are empty or very short on a JavaScript-rendered site.**

The page content may not be rendered when Playwright first loads it. Add the main content selector to `excludeSelectors`'s inverse — or simply let the actor wait by setting `navigationTimeoutSecs` to a higher value (default: 30 s).

**I'm seeing nav menu text in every chunk.**

Add the offending element's CSS selector to `excludeSelectors` (e.g. `[".custom-nav", "#site-sidebar"]`). The actor already strips the most common patterns automatically.

**Versioned docs produce duplicate chunks.**

Use `versionFilter` to restrict crawling to one version (e.g. `/v3/`), or leave `deduplicateContent: true` to drop identical chunks regardless of which version URL they came from.

# Actor input Schema

## `startUrls` (type: `array`):

Root URL(s) of the documentation site to crawl.

## `urlPatterns` (type: `array`):

Glob patterns to restrict which URLs get crawled. E.g. /docs/**, /api/**. Leave empty to crawl all same-domain URLs.

## `maxDepth` (type: `integer`):

Link levels to follow from each start URL (0 = start pages only).

## `maxRequestsPerCrawl` (type: `integer`):

Hard cap on pages crawled. 0 = unlimited (capped at 200 by the actor).

## `chunkStrategy` (type: `string`):

How to split page content into chunks.

## `chunkSizeTokens` (type: `integer`):

Target token count per chunk. 512 works well for most embedding models.

## `chunkOverlapTokens` (type: `integer`):

Tokens of overlap between adjacent chunks (for fixed_tokens strategy).

## `outputFormat` (type: `string`):

Schema of each dataset record. Use 'markdown' for generic RAG pipelines; framework-specific formats for LangChain / LlamaIndex / Haystack; 'json-chunks' for direct Pinecone / Chroma / pgvector inserts; 'text' for plain-text pipelines.

## `chunkSize` (type: `integer`):

Shorthand for Chunk Size (tokens). Set this to split pages into vector-ready chunks of ~N tokens with full metadata (source URL, title, section header). 500 is a solid default for most embedding models (text-embedding-3-small, nomic-embed-text, etc.). Overrides chunkSizeTokens when set.

## `excludeSelectors` (type: `array`):

Additional CSS selectors to strip before extraction (e.g. .announcement-bar, #feedback-widget).

## `versionFilter` (type: `string`):

Only crawl URLs whose path contains this string (e.g. /v2/ or /stable/). Leave empty for all versions.

## `deduplicateContent` (type: `boolean`):

Skip chunks whose content hash was already seen in this run. Prevents duplicates from paginated or mirrored docs.

## `pdfHandling` (type: `string`):

What to do when the crawler encounters a .pdf URL.

## `awsTextractRegion` (type: `string`):

AWS region for Textract and DynamoDB. Applies to all AWS services.

## `awsS3Bucket` (type: `string`):

S3 bucket used as temp staging for Textract jobs. Required when PDF Handling = AWS Textract. PDFs are uploaded, processed, then deleted.

## `useDeltaTracking` (type: `boolean`):

Store a content-hash fingerprint per URL in DynamoDB. On re-runs, unchanged pages are skipped entirely — reducing Apify credits and vector-DB upsert costs by up to 80% on stable doc sites.

## `dynamodbTableName` (type: `string`):

Table name for delta tracking. Created automatically on first run (PAY_PER_REQUEST billing, 90-day TTL). Required if Incremental Crawling is enabled.

## `notifyWebhookUrl` (type: `string`):

POST the dataset ID and item count to this URL when the run finishes.

## `useProxy` (type: `boolean`):

Route requests through Apify proxy. Usually not needed for public doc sites.

## `proxyConfiguration` (type: `object`):

Advanced proxy settings.

## `maxConcurrency` (type: `integer`):

Parallel browser pages.

## `navigationTimeoutSecs` (type: `integer`):

Per-page timeout before marking as failed.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev/docs/introduction"
    }
  ],
  "urlPatterns": [],
  "maxDepth": 5,
  "maxRequestsPerCrawl": 200,
  "chunkStrategy": "heading_boundary",
  "chunkSizeTokens": 512,
  "chunkOverlapTokens": 50,
  "outputFormat": "markdown",
  "excludeSelectors": [],
  "deduplicateContent": true,
  "pdfHandling": "skip",
  "awsTextractRegion": "us-east-1",
  "useDeltaTracking": false,
  "dynamodbTableName": "rag-docs-scraper-delta",
  "useProxy": false,
  "maxConcurrency": 5,
  "navigationTimeoutSecs": 30
}
```

# Actor output Schema

## `page_content` (type: `string`):

Cleaned text content of the chunk.

## `metadata` (type: `string`):

JSON object with source URL, heading_path, chunk_index, chunk_total, token_count, content_hash, scraped_at.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev/docs/introduction"
        }
    ],
    "urlPatterns": [],
    "excludeSelectors": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("lernowa_ai_labs/rag-docs-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://crawlee.dev/docs/introduction" }],
    "urlPatterns": [],
    "excludeSelectors": [],
}

# Run the Actor and wait for it to finish
run = client.actor("lernowa_ai_labs/rag-docs-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev/docs/introduction"
    }
  ],
  "urlPatterns": [],
  "excludeSelectors": []
}' |
apify call lernowa_ai_labs/rag-docs-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,lernowa_ai_labs/rag-docs-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/9kOkEMe0j1XJm9OME/builds/UehuAbnRnPSh4aXVo/openapi.json