# Website to LLM Markdown – RAG Chunks & Token Counts (`fast_data_asap/website-to-llm-markdown`) Actor

Crawl any website or sitemap and get clean LLM-ready Markdown per page: boilerplate removed, split into heading-aware RAG chunks with token counts for vector databases, embeddings and AI agents. Fast, no browser, respects robots.txt. Pay per page.

- **URL**: https://apify.com/fast_data_asap/website-to-llm-markdown.md
- **Developed by:** [Fast Data](https://apify.com/fast_data_asap) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.50 / 1,000 page converteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to LLM-ready Markdown – RAG Chunks & Token Counts

Turn any public website, docs site or sitemap into **clean Markdown for LLMs**, already **split into heading-aware RAG chunks with token counts**. Navigation, footers, cookie banners and other boilerplate are removed. Load the output straight into Pinecone, Qdrant, Weaviate, pgvector, Chroma, LangChain or LlamaIndex.

**$1.50 per 1,000 pages. You pay only for pages that were converted.** Failed, empty or blocked pages are listed with a status and are free.

#### Why this crawler

- ✅ **Chunked and token-counted out of the box**: every page comes with a `chunks` array (`text`, `tokenCount`, `headingPath`, `chunkId`), so you don't need a separate splitting step
- ✅ **Real tokenizer counts**: `o200k_base` (GPT-4o, GPT-5) or `cl100k_base` (GPT-4, `text-embedding-3`), not a rough guess
- ✅ **Fast and cheap**: plain HTTP, no headless browser. 3 pages finish in about 5 seconds, compute costs are tiny
- ✅ **Smart chunking**: chunks follow headings and paragraphs, code blocks stay intact, optional token overlap
- ✅ **Polite**: respects `robots.txt`, limited concurrency, own user agent
- ✅ **Sitemap support**: pass `sitemap.xml` (also sitemap indexes and `.xml.gz`) to convert a whole site without crawling

**Who it's for:** developers building RAG chatbots and AI agents, teams that want their docs in a vector database, and anyone feeding web pages to ChatGPT, Claude or Gemini.

**When to use something else:** pages that only render content with JavaScript (single-page apps) need a browser-based crawler. This actor is built for the large majority of docs sites, blogs, help centers and marketing sites that serve HTML.

### Input

| Field | Default | Description |
|---|---|---|
| `startUrls` | `https://docs.apify.com/` | Start pages, or a sitemap URL |
| `maxPages` | 3 | Maximum pages converted per run |
| `maxDepth` | 3 | Link hops to follow from the start URLs (0 = start URLs / sitemap pages only) |
| `sameDomain` | true | Only follow links on the start URL's host |
| `includeUrlPatterns` / `excludeUrlPatterns` | – | Globs like `https://docs.example.com/guides/**` or plain substrings like `/blog/` |
| `chunkSize` | 500 | Maximum tokens per chunk |
| `chunkOverlap` | 0 | Tokens repeated from the previous chunk |
| `tokenizer` | `o200k_base` | `o200k_base` or `cl100k_base` |
| `includeLinks`, `includeTables`, `includeChunks` | false, true, true | Output options |
| `respectRobotsTxt` | true | Skip URLs disallowed by robots.txt |
| `proxyConfiguration` | off | Optional proxy for sites that rate-limit |

```json
{
  "startUrls": [{ "url": "https://docs.apify.com/" }],
  "maxPages": 50,
  "includeUrlPatterns": ["https://docs.apify.com/platform/**"],
  "chunkSize": 400,
  "chunkOverlap": 40
}
```

### Output

One dataset item per page. Real example from a test run (Markdown shortened):

```json
{
  "url": "https://docs.apify.com/actors",
  "depth": 1,
  "status": "ok",
  "httpStatus": 200,
  "title": "Actors | Platform | Apify Documentation",
  "description": "Learn how to develop, run and share serverless cloud programs. Create your own web scraping and automation tools and publish them on the Apify platform.",
  "language": "en",
  "canonicalUrl": "https://docs.apify.com/actors",
  "wordCount": 229,
  "tokenCountEstimate": 294,
  "tokenizer": "o200k_base",
  "chunkCount": 1,
  "markdown": "# Actors\n\nActors are serverless cloud programs that take a structured JSON input, perform a task (web scraping, browser automation, data processing, and more), and optionally produce a structured output. ...\n\n## Actor components\n\n- *Dockerfile* which specifies where the Actor's source code is ...",
  "chunks": [
    {
      "chunkIndex": 0,
      "chunkId": "https://docs.apify.com/actors#chunk-0",
      "headingPath": "Actors",
      "headings": ["Actors"],
      "text": "# Actors\n\nActors are serverless cloud programs ...",
      "tokenCount": 294,
      "charCount": 1393
    }
  ]
}
```

| Field | Description |
|---|---|
| `url`, `loadedUrl`, `canonicalUrl`, `depth` | Requested URL, URL after redirects, canonical link, crawl depth |
| `status` | `ok`, `failed`, `no_content` or `blocked_by_robots_txt` (only `ok` is charged) |
| `title`, `description`, `language`, `author`, `publishedDate`, `siteName` | Page metadata |
| `markdown` | Main content as Markdown, boilerplate removed |
| `wordCount`, `charCount`, `tokenCountEstimate` | Size of the Markdown |
| `chunks[]` | `chunkIndex`, `chunkId`, `headingPath` (e.g. `Guides > Install > Linux`), `headings`, `text`, `tokenCount`, `charCount` |

The dataset has two views: **Overview** (one row per page) and **RAG chunks** (one row per chunk).

### Pricing

Pay per event: **$0.0015 per page converted** ($1.50 per 1,000 pages). Pages that fail, are empty or are blocked by robots.txt cost nothing. Because no browser is used, platform compute usage stays very low.

### Use it via API, n8n, Make or MCP

**API (sync, returns items):**

```bash
curl -X POST "https://api.apify.com/v2/acts/fast_data_asap~website-to-llm-markdown/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://docs.apify.com/"}],"maxPages":10,"chunkSize":500}'
```

**Python:**

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")
run = client.actor("fast_data_asap/website-to-llm-markdown").call(run_input={
    "startUrls": [{"url": "https://docs.apify.com/"}], "maxPages": 10,
})
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    for chunk in page.get("chunks", []):
        print(chunk["chunkId"], chunk["tokenCount"], chunk["headingPath"])
```

**n8n / Make / Zapier:** use the Apify integration, choose the actor `fast_data_asap/website-to-llm-markdown` and map the `chunks` array into your vector store node.

**MCP (Claude, Cursor, other AI agents):** add `https://mcp.apify.com?tools=fast_data_asap/website-to-llm-markdown` as an MCP server so your agent can read whole websites as Markdown on demand.

### FAQ

**How are tokens counted?** With the `tiktoken` encoding you choose. For Claude, Gemini and Llama the counts are a close estimate.

**Does it follow robots.txt?** Yes, by default. You can switch this off for your own sites.

**Can I convert only part of a site?** Yes: use `includeUrlPatterns`, `excludeUrlPatterns` and `maxDepth`, or pass a sitemap.

**Is it legal?** The actor only reads publicly available pages without logging in. Make sure your use of the content respects the site's terms and copyright.

# Actor input Schema

## `startUrls` (type: `array`):

Pages to start crawling from. A sitemap URL (e.g. https://example.com/sitemap.xml) is expanded into its page URLs.

## `maxPages` (type: `integer`):

Maximum number of pages converted to Markdown in this run.

## `maxDepth` (type: `integer`):

How many link hops to follow from the start URLs. 0 = only the start URLs (or sitemap pages).

## `sameDomain` (type: `boolean`):

Only follow links on the same host as the start URL (www. is ignored).

## `includeUrlPatterns` (type: `array`):

Optional. Only crawl URLs matching at least one pattern. Glob (e.g. https://docs.apify.com/platform/\*\*) or plain substring (e.g. /docs/).

## `excludeUrlPatterns` (type: `array`):

Optional. Skip URLs matching any pattern, e.g. **/changelog/** or ?page=.

## `chunkSize` (type: `integer`):

Target maximum tokens per RAG chunk. Chunks follow headings and paragraphs, never cut code blocks unless they are larger than this.

## `chunkOverlap` (type: `integer`):

Tokens of the previous chunk repeated at the start of the next one within the same section. 0 = no overlap.

## `tokenizer` (type: `string`):

Token counting encoding. o200k_base = GPT-4o / GPT-4.1 / GPT-5 family, cl100k_base = GPT-4 / GPT-3.5 / text-embedding-3. Counts are a close estimate for Claude, Gemini and Llama too.

## `includeLinks` (type: `boolean`):

Keep hyperlinks as [text](url). Off gives cleaner text for embeddings.

## `includeTables` (type: `boolean`):

Convert HTML tables to Markdown tables.

## `includeChunks` (type: `boolean`):

Add the `chunks` array to each page.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs that the site's robots.txt disallows.

## `maxConcurrency` (type: `integer`):

Parallel requests. Keep low to be polite to small sites.

## `proxyConfiguration` (type: `object`):

Optional. By default pages are fetched directly (fast and free). Enable a proxy if a site rate-limits you.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/"
    }
  ],
  "maxPages": 3,
  "maxDepth": 3,
  "sameDomain": true,
  "chunkSize": 500,
  "chunkOverlap": 0,
  "tokenizer": "o200k_base",
  "includeLinks": false,
  "includeTables": true,
  "includeChunks": true,
  "respectRobotsTxt": true,
  "maxConcurrency": 5,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/"
        }
    ],
    "maxPages": 3
};

// Run the Actor and wait for it to finish
const run = await client.actor("fast_data_asap/website-to-llm-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/" }],
    "maxPages": 3,
}

# Run the Actor and wait for it to finish
run = client.actor("fast_data_asap/website-to-llm-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/"
    }
  ],
  "maxPages": 3
}' |
apify call fast_data_asap/website-to-llm-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,fast_data_asap/website-to-llm-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/4pBsieINBwWKZjRMG/builds/6NexJvw240VRmdLGl/openapi.json