# Website to Markdown Crawler - RAG-Ready Chunks for LLMs (`sonic_scraper/ai-rag-web-crawler`) Actor

Website to Markdown crawler for LLMs: turn docs, blogs and help centers into clean Markdown with URL, title and description, or paragraph-aware overlapping chunks for RAG and vector databases. Boilerplate removed. $2 per 1,000 pages, no AI key.

- **URL**: https://apify.com/sonic_scraper/ai-rag-web-crawler.md
- **Developed by:** [Group Oject](https://apify.com/sonic_scraper) (community)
- **Categories:** AI, Agents, Developer tools
- **Features:** Limited permissions
- **Stats:** 2 total users, 0 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.40 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown Crawler - RAG-Ready Chunks for LLMs

Crawl docs sites, blogs, and help centers and convert every page into **clean Markdown** with its URL, title, and meta description. Get one row per page, or switch on chunking to get **overlapping, embedding-ready chunks** for RAG pipelines, vector databases, chatbots, and AI agents. Fast HTTP crawler, no AI API key needed. **$2.00 per 1,000 rows.**

![Website to Markdown Crawler output in Apify: url, title, chunk #, of, chars, content (markdown), depth](https://api.apify.com/v2/key-value-stores/lwQu1lCMCLQIIhSMQ/records/ai-rag-web-crawler-output.png)

*Real output from a default run of Website to Markdown Crawler in Apify Console.*

### What can this website to Markdown crawler do?

- **Convert web pages to Markdown** - headings, lists, links, tables, and code blocks preserved.
- **Remove boilerplate** - navigation, headers, footers, cookie banners, scripts, and similar clutter are stripped; the densest main-content container (`article`, `main`, `#content`...) is preferred.
- **Crawl a whole site or one section** - set crawl depth, page limit, same-domain restriction, and include/exclude URL globs.
- **Chunk for RAG** - paragraph-aware chunks with configurable size, overlap, and minimum length, each tagged with `chunkIndex` and `chunkCount`.
- **Keep source attribution** - every row carries the page URL and title, so your chatbot can cite sources.
- **Optional cleaned HTML** of the main content alongside the Markdown.

### What data can I extract?

| Field | Description |
|---|---|
| `url` | Source page URL |
| `title` | Page title |
| `description` | Meta description |
| `content` | Markdown of the page (or of one chunk) |
| `contentChars` | Length of `content` |
| `chunkIndex`, `chunkCount` | Position of the chunk within its page (0 / 1 when chunking is off) |
| `html` | Cleaned main-content HTML (only when `saveHtml` is on) |
| `depth` | Link hops from the start URL |
| `crawledAt` | Timestamp |

The key-value store `SUMMARY` record reports pages crawled, pages failed, total chunks, total characters, and average chunk size.

### How much does it cost?

Pay-per-event: **$0.002 per dataset row** plus a tiny Actor-start fee.

- **Chunking off:** one row per page = **$2.00 per 1,000 pages**.
- **Chunking on:** one row per chunk. A typical docs page produces a few chunks at the default 1,000 characters, so cost per page rises accordingly. Increase `chunkSize` to reduce rows, or turn chunking off and chunk in your own pipeline.

Check the Pricing tab for current rates.

### How to convert a website to Markdown

1. Add one or more start URLs (for example `https://docs.example.com/`).
2. Set `maxCrawlPages` and `maxCrawlDepth` (0 = only the start URLs).
3. Optionally limit the crawl with `includeUrlGlobs` such as `https://docs.example.com/guides/*`.
4. Choose whole pages (`chunkContent: false`) or RAG chunks (`chunkContent: true`).
5. Click **Start** and export JSON, JSONL, or CSV - or load the dataset straight into LangChain or LlamaIndex.

### Input example

```json
{
  "startUrls": ["https://docs.apify.com/"],
  "maxCrawlPages": 200,
  "maxCrawlDepth": 2,
  "sameDomainOnly": true,
  "includeUrlGlobs": ["https://docs.apify.com/platform/*"],
  "chunkContent": true,
  "chunkSize": 1500,
  "chunkOverlap": 150
}
```

### Output example

```json
{
  "url": "https://docs.apify.com/platform/actors",
  "title": "Actors | Platform | Apify Documentation",
  "description": "Learn how to develop, run and share serverless cloud programs.",
  "chunkIndex": 0,
  "chunkCount": 3,
  "content": "# Actors\n\nActors are serverless cloud programs that can perform anything from a simple action...",
  "contentChars": 1432,
  "depth": 1,
  "crawledAt": "2026-10-25T10:00:00.000Z"
}
```

Illustrative output shape.

### Use cases

- **RAG knowledge bases** - index your docs or help center for a support chatbot.
- **Custom GPTs and AI agents** - give an agent a clean, cited copy of a website.
- **Vector database ingestion** - Pinecone, Qdrant, Weaviate, pgvector, Chroma.
- **Content migration** - move a blog or docs site to a Markdown-based CMS.
- **Competitive research** - read a competitor's docs or pricing pages as plain Markdown.

### Integrations and API

Use the Apify API, LangChain and LlamaIndex Apify loaders, Make, Zapier, n8n, or the Apify MCP server.

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("sonic_scraper/ai-rag-web-crawler").call(run_input={
    "startUrls": ["https://docs.example.com/"],
    "maxCrawlPages": 100,
    "chunkContent": True,
})
docs = [(row["url"], row["content"]) for row in client.dataset(run["defaultDatasetId"]).iterate_items()]
```

### FAQ

**Is this a Firecrawl alternative?** For server-rendered sites such as docs, blogs and help centers, yes: it converts a website or URL to clean Markdown and RAG-ready chunks, paid per page and with no AI key. It is an HTTP crawler, so pages that only render with JavaScript need a browser-based crawler instead.

**Does it render JavaScript?** No. It is an HTTP crawler (Cheerio) built for server-rendered sites such as most docs platforms, blogs, and help centers. Single-page apps that load content with JavaScript may return little text.

**Does it need an OpenAI or other AI key?** No. Extraction and chunking are deterministic.

**How is this different from Apify's Website Content Crawler?** Website Content Crawler is a general-purpose crawler with browser rendering and many options. This Actor is a lightweight HTTP-only crawler with built-in chunking and a simple per-row price.

**Can it get blocked?** Some sites block datacenter IPs; enable Apify Proxy in `proxyConfiguration`.

### Related Actors

- [Technical SEO Audit Crawler](https://apify.com/sonic_scraper/technical-seo-audit-crawler) - audit the same pages for SEO issues.
- [Contact Details Scraper](https://apify.com/sonic_scraper/b2b-website-intelligence-scraper) - extract emails, phones, and tech stack from company sites.

# Changelog

This Actor's version history is a separate document: https://apify.com/sonic_scraper/ai-rag-web-crawler/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

URLs to start crawling from. Accepts plain URLs or { "url": "..." } objects.

## `maxCrawlPages` (type: `integer`):

Maximum number of pages to crawl in total.

## `maxCrawlDepth` (type: `integer`):

How many link-hops to follow from the start URLs. 0 = only the start URLs.

## `sameDomainOnly` (type: `boolean`):

Only follow links on the same domain(s) as the start URLs.

## `includeUrlGlobs` (type: `array`):

Only crawl URLs matching these glob patterns (e.g. https://site.com/docs/\*). Empty = no include filter.

## `excludeUrlGlobs` (type: `array`):

Skip URLs matching these glob patterns (e.g. *.pdf, https://site.com/blog/*).

## `chunkContent` (type: `boolean`):

Split each page into overlapping chunks (one dataset row per chunk) — ready for embeddings. Off = one row per page.

## `chunkSize` (type: `integer`):

Target characters per chunk.

## `chunkOverlap` (type: `integer`):

Characters of context carried between consecutive chunks.

## `minChunkChars` (type: `integer`):

Drop chunks shorter than this many characters.

## `saveHtml` (type: `boolean`):

Also include the cleaned HTML of the main content in each row.

## `maxConcurrency` (type: `integer`):

How many pages to crawl in parallel.

## `proxyConfiguration` (type: `object`):

Optional. Use Apify Proxy to crawl sites that block datacenter IPs.

## `debugMode` (type: `boolean`):

Verbose logging.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/"
    }
  ],
  "maxCrawlPages": 50,
  "maxCrawlDepth": 1,
  "sameDomainOnly": true,
  "chunkContent": true,
  "chunkSize": 1000,
  "chunkOverlap": 100,
  "minChunkChars": 50,
  "saveHtml": false,
  "maxConcurrency": 10,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "debugMode": false
}
```

# Actor output Schema

## `chunks` (type: `string`):

One row per chunk: url, title, chunk index, and LLM-ready Markdown content.

## `summary` (type: `string`):

Pages crawled/failed, total chunks, average chunk size, settings.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/"
        }
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("sonic_scraper/ai-rag-web-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/" }],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("sonic_scraper/ai-rag-web-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call sonic_scraper/ai-rag-web-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,sonic_scraper/ai-rag-web-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/LoEsRxGcgYYlgMVzX/builds/rMPugQOXrw5J8RSIg/openapi.json
