# LLM Website Crawler: Clean Markdown & RAG Chunks (`zaiq/llm-website-crawler`) Actor

Crawl any website and get every page's main content as clean Markdown for ChatGPT, RAG and AI agents: no menus, footers or cookie banners, with title, description, headings and dates, plus ready-to-embed chunks. Sitemaps, JavaScript pages, include and exclude patterns. $1 per 1,000 pages.

- **URL**: https://apify.com/zaiq/llm-website-crawler.md
- **Developed by:** [Zaiq](https://apify.com/zaiq) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.80 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown Crawler for LLMs, RAG and AI agents ($1 per 1,000 pages)

Crawl **any website** and get **every page's main content as clean Markdown**: no menus, footers, cookie banners or scripts, with **code blocks kept** (language and indentation), **tables**, **links and images as full URLs**, plus each page's **title, description, language, dates and headings**.

Turn on **chunks** and each page is split **on its headings** into **ready-to-embed pieces** with their heading path, for **RAG, vector databases, custom GPTs and AI agents**. It reads the site's **sitemaps**, opens **JavaScript pages** in Chrome only when needed, and respects **robots.txt**.

### What you get

| Field | What it holds |
|---|---|
| Page rows | `url`, `title`, `description`, `language`, `canonicalUrl`, `author`, `publishedAt`, `modifiedAt`, `imageUrl`, `depth`, `referrer`, `statusCode`, `headings`, `wordCount`, `markdown`; optional `text`, `html` |
| Chunks (optional) | `chunks` inside each page, or one row per chunk: `chunkIndex`, `headings`, `text`, `characters`, `chunkCount` |
| SUMMARY | pages delivered, failed, left out by robots.txt, duplicates, pages found in sitemaps, pages opened in Chrome, what was charged |

### What people use it for

- **RAG and vector search:** a docs site or help centre as chunks to embed
- **Custom GPTs and assistants:** your whole website as knowledge files
- **AI agents:** read a site's pages as Markdown in one call
- **Content audits and migrations:** every page's text, headings and metadata

### How to use it

1. Add a **start URL**: it crawls that site under that path (start at `/docs/` for the docs only).
2. Set **Most pages** and, for RAG, **Chunks**.
3. Click **Start** and download the pages as JSON, CSV or Excel, or read them by API.

### Input

| Field | Example | Notes |
|---|---|---|
| Start URLs | `https://docs.example.com/guide/` | Crawls under this path |
| Most pages | `100` | Up to 100,000 |
| Use the site's sitemaps | on | Finds pages no link points to |
| JavaScript pages | `auto` | Chrome only when the HTML is empty |
| Chunks for RAG | `none` | `per-page` or `rows` |
| Chunk size | `2000` | Characters, overlap 200 |
| Only / skip URLs matching | `*/blog/*` | Wildcards |
| Content selector | `article` | Optional: the part to keep |

### Output

Each result is one row in the run's dataset (download it as JSON, CSV, Excel, XML or HTML, or read it through the API). An example row (values shortened):

````json
{
  "url": "https://docs.example.com/tutorial/controlflow.html",
  "title": "4. More Control Flow Tools",
  "description": "As well as the while statement just introduced, the language uses a few more…",
  "language": "en",
  "depth": 1,
  "headings": [
    "4. More Control Flow Tools",
    "4.1. `if` Statements",
    "4.2. `for` Statements"
  ],
  "wordCount": 5915,
  "markdown": "# 4. More Control Flow Tools\n\nAs well as the [`while`](https://docs.example.com/reference/compound_stmts.html#while) statement…\n\n## 4.1. `if` Statements\n\n```python\n>>> x = int(input(\"Please enter an integer: \"))\n>>> if x < 0:\n...     x = 0\n```…",
  "chunks": [
    {
      "chunkIndex": 0,
      "headings": [
        "4. More Control Flow Tools"
      ],
      "text": "# 4. More Control Flow Tools…",
      "characters": 1475
    }
  ],
  "crawledAt": "2026-10-17T18:45:15Z"
}
````

From runs on 17 October 2026 (a programming language's documentation and a company blog, addresses replaced): 12 tutorial pages in 7 seconds with every code example fenced and indented and the navigation and heading anchors removed; on the blog, the crawl stayed under /blog, found 24 posts in its sitemaps and wrote 8 of them as 33 chunks. A record named `SUMMARY` in the run's key-value store counts every page's outcome and what was charged.

### Pricing

- **$0.001 per page** delivered with its content ($1 per 1,000 pages); chunks cost nothing extra
- Free: pages that fail, are blocked, are left out by robots.txt or your patterns, or repeat another page
- No AI usage: the conversion runs without a language model

What typical jobs cost at the list price:

| Job | Cost |
|---|---|
| A 200-page docs site | $0.20 |
| A blog's 50 latest posts as chunks | $0.05 |
| 10,000 pages | $10 |

On Apify's Scale and Business plans the per-event prices are lower (Store discounts). Set **Maximum cost per run** and the Actor stops cleanly when it is reached: you never pay more than that.

### Tips

- Start at the section you need (`/docs/`, `/help/`) and the crawl stays inside it; switch off Stay under the start URL's path for the whole site.
- Choose one row per chunk to load the dataset straight into a vector database.
- Use Skip URLs matching for tag, search and login pages (`*/tag/*`, `*?page=*`).
- Schedule it weekly to keep an AI's knowledge of a site fresh.

### Use it through the API

**Python**

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("zaiq/llm-website-crawler").call(run_input={"startUrls": [{"url": "https://docs.example.com/"}], "maxPages": 200, "chunking": "rows"})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row)
```

**JavaScript**

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('zaiq/llm-website-crawler').call({"startUrls": [{"url": "https://docs.example.com/"}], "maxPages": 200, "chunking": "rows"});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
```

**HTTP (one call, results back)**

```bash
curl -X POST 'https://api.apify.com/v2/acts/zaiq~llm-website-crawler/run-sync-get-dataset-items?token=<YOUR_APIFY_TOKEN>' \
  -H 'Content-Type: application/json' \
  -d '{"startUrls": [{"url": "https://docs.example.com/"}], "maxPages": 200, "chunking": "rows"}'
```

### Integrations and schedules

- **Schedules:** run it every hour, day or week from the Apify Console (Schedules) and keep a history of results.
- **Make, Zapier, n8n, Google Sheets, Slack, Airtable:** connect it through Apify's integrations, or send each finished run to a **webhook**.
- **Datasets:** results stay in the run's dataset; export them or read them by API whenever you need them.

### Use it with AI agents (MCP)

Add Apify's MCP server (`https://mcp.apify.com`) to Claude, ChatGPT, Cursor, VS Code or any MCP client. The agent can find this Actor and run it for you: ask it for "read the docs at this address and give me every page as Markdown". AI agents can also pay for runs directly through Apify's agentic payments.

### Limits

- It connects directly and through Apify's datacenter proxies: a site that blocks both is reported as failed (free). The Website Content Crawler to Markdown by Country reads such sites through residential connections.
- PDFs, images and other files are skipped.
- Pages behind a login are not read.
- A page's Markdown is cut at 400,000 characters.

### FAQ

**Is it an alternative to Apify's Website Content Crawler?**

Yes, with a fixed price per page instead of compute time, chunks on headings and code blocks kept with their language.

**Does it read JavaScript sites?**

Yes: a page whose HTML carries no content opens in Chrome (auto), or every page with Always.

**Does it follow robots.txt?**

Yes by default; it can be switched off for your own sites.

**Can I get plain text or HTML too?**

Yes: switch on Also plain text or Also the page's HTML.

### Related Actors from Zaiq

- [ai-web-scraper](https://apify.com/zaiq/ai-web-scraper): the data you describe from any page, as JSON
- [website-crawler-by-country](https://apify.com/zaiq/website-crawler-by-country): sites read from a chosen country through residential connections
- [bulk-ai-runner](https://apify.com/zaiq/bulk-ai-runner): an AI prompt on every page
- [company-research-agent](https://apify.com/zaiq/company-research-agent): company profiles from websites

### Support

Questions, a site that changed, or a field you need? Open an issue on this Actor's **Issues** tab: we answer there, usually within a day.

***

Also found as: website content crawler, website to markdown, rag crawler, llm crawler, web crawler for ai, docs to markdown, crawl website for chatgpt, markdown scraper, vector database ingestion.

# Actor input Schema

## `startUrls` (type: `array`):

Where the crawl starts. It follows links on the same site and under the same path (start at https://example.com/docs/ to crawl only the docs).

## `maxPages` (type: `integer`):

The crawl stops after this many pages delivered (in all start URLs).

## `maxDepth` (type: `integer`):

0 reads only the start pages (and their sitemaps when on).

## `useSitemaps` (type: `boolean`):

Also find pages in the sitemaps named in robots.txt or at /sitemap.xml (pages no link points to as well).

## `renderJs` (type: `string`):

Most pages carry their content in the HTML; a page built by JavaScript is opened in Chrome to read it.

## `chunking` (type: `string`):

Split each page's Markdown on its headings into chunks of the size below, each with its heading path. A page is charged once however many chunks it has.

## `chunkSize` (type: `integer`):

The longest a chunk may be, in characters (2,000 is about 400 words).

## `chunkOverlap` (type: `integer`):

Repeated at the cut when a section longer than the chunk size is split.

## `includeUrlGlobs` (type: `array`):

Patterns with \* wildcards, e.g. https://example.com/blog/\*. Empty: every page in scope.

## `excludeUrlGlobs` (type: `array`):

e.g. *?page=*, */tag/*, */login*

## `includeSubdomains` (type: `boolean`):

Also follow links to blog.example.com, docs.example.com and so on.

## `stayUnderStartPath` (type: `boolean`):

Start at /docs/ and only /docs/ pages are read. Off: the whole site.

## `respectRobotsTxt` (type: `boolean`):

Leave a page out when the site's robots.txt asks crawlers to. Switch off only for your own sites.

## `contentSelector` (type: `string`):

A CSS selector for the part of the page to keep, e.g. article or main .content. Empty: found automatically.

## `removeSelectors` (type: `array`):

CSS selectors to drop before conversion, e.g. .newsletter-signup, #comments.

## `includeText` (type: `boolean`):

Add each page's content as plain text (no Markdown) too.

## `includeHtml` (type: `boolean`):

Add each page's full HTML too (larger rows).

## `maxConcurrency` (type: `integer`):

How many pages are read at the same time.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/"
    }
  ],
  "maxPages": 10,
  "maxDepth": 20,
  "useSitemaps": true,
  "renderJs": "auto",
  "chunking": "none",
  "chunkSize": 2000,
  "chunkOverlap": 200,
  "includeSubdomains": false,
  "stayUnderStartPath": true,
  "respectRobotsTxt": true,
  "includeText": false,
  "includeHtml": false,
  "maxConcurrency": 8
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset: the results.

## `summary` (type: `string`):

What each input returned, and notes.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.python.org/3/tutorial/"
        }
    ],
    "maxPages": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("zaiq/llm-website-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.python.org/3/tutorial/" }],
    "maxPages": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("zaiq/llm-website-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/"
    }
  ],
  "maxPages": 10
}' |
apify call zaiq/llm-website-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,zaiq/llm-website-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/NcXQsKa2J8YQBEXlW/builds/E6ny7MDmmNZHa7Pdo/openapi.json