# Website Content Crawler for AI - Markdown & RAG Chunks (`kantolabs/website-content-crawler-ai`) Actor

Crawl a website to clean Markdown, text and token-counted RAG chunks for LLMs, vector DBs, n8n and Make. Sitemaps, robots.txt, JS pages, linked PDF/DOCX, llms.txt, change-only re-crawls. Flat price per page.

- **URL**: https://apify.com/kantolabs/website-content-crawler-ai.md
- **Developed by:** [Kanto Labs](https://apify.com/kantolabs) (community)
- **Categories:** AI, Developer tools, Integrations
- **Features:** Limited permissions
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.13 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does Website Content Crawler for AI do?

**Website Content Crawler for AI** is a **website content crawler** that turns a whole website into **clean Markdown, plain text and RAG-ready chunks** for LLMs, vector databases, chatbots and AI agents. Give it the start of a docs site, blog, help centre or company site and it follows the links and the **sitemap**, keeps the **main content** of every page (navigation, headers, footers, sidebars, cookie banners, ads and share buttons are removed), and returns one row per page with the **Markdown**, **text**, **title, description, language**, a **content hash** and **heading-aware chunks with token counts**.

You pay a **flat price per page**, so you know the cost before you press Start.

### Why use this website content crawler?

- **Flat, predictable price per page.** No compute-unit maths, no surprise bills from slow pages, retries or a big memory setting. **$0.15 per 1,000 pages** (plain HTTP), **$0.90 per 1,000 pages** that need a headless browser.
- **RAG-ready output.** Every page comes pre-split into **heading-aware chunks** (default 512 tokens, 64 tokens overlap, OpenAI `cl100k_base` token counts). Each chunk carries the page URL, title and **heading path** (`["Guide", "Install", "Windows"]`), so you can embed it straight away.
- **Linked documents too.** Turn on **Convert linked documents** and the **PDF, DOCX, PPTX and XLSX** files linked from the pages become Markdown + chunks as well.
- **`llms.txt` and the whole site in one file.** Each run writes `llms.txt` (a page index), `llms-full.txt` (every page's Markdown in one file) and `chunks.jsonl` (all chunks, one per line) to the key-value store.
- **Cheap re-crawls: only changed pages.** Point a new run at the dataset of an earlier run: pages that did not change (same content hash, or `304 Not Modified`) come back as free `unchanged` rows. You pay only for new and changed pages.
- **Duplicates removed for free.** Same canonical URL, identical text or near-identical text (SimHash: print views, tracking-parameter and sort variants) become free `duplicate` rows that point to the first copy.
- **JavaScript sites.** Fast plain HTTP by default; pages that only show content after JavaScript runs are automatically re-rendered in **headless Chromium**.
- **Polite and safe.** robots.txt is respected by default (with `Crawl-delay`), the crawler identifies itself as `KantoCrawler`, and **failed, blocked, robots-disallowed, duplicate and unchanged pages are never charged**. Bot walls, CAPTCHAs and logins are never bypassed.

### How to crawl a website to Markdown

1. Put the start of the site in **Start URLs**, e.g. `https://docs.example.com/` (pages under the same path are followed).
2. Set **Max pages** (and, if you want, include / exclude URL globs and a max depth).
3. Keep **Crawler type = auto** (HTTP, browser only when needed). Choose **http** for the lowest price or **browser** for app-like sites.
4. Optional: **Convert linked documents**, a different **chunk size**, or **Only changed pages since this dataset** for a re-crawl.
5. Click **Start**. Pages appear in the **Pages** view; the **OUTPUT** record links `llms.txt`, `llms-full.txt` and `chunks.jsonl`.

### Input example

```json
{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "maxPages": 500,
  "crawlerType": "auto",
  "excludeUrlGlobs": ["https://docs.example.com/**/changelog/**"],
  "chunkSize": 512,
  "chunkOverlap": 64,
  "convertLinkedDocuments": true
}
```

### Output example

One row per page (Markdown, text and chunks shortened):

```json
{
  "url": "https://www.joelonsoftware.com/2018/04/13/gamification/",
  "status": "ok",
  "type": "page",
  "renderedWith": "http",
  "httpStatus": 200,
  "depth": 1,
  "title": "A Dusting of Gamification – Joel on Software",
  "description": "I had to think for a minute to realize that Stack Overflow has \"gamification\" too...",
  "language": "en-US",
  "publishedAt": "2018-04-13T13:40:21+00:00",
  "markdown": "# A Dusting of Gamification\n\nAround 2010 the success of Stack Overflow had led us into some conversations with VCs...",
  "text": "A Dusting of Gamification\n\nAround 2010 the success of Stack Overflow...",
  "wordCount": 1235,
  "tokenCount": 1690,
  "contentHash": "bb8a180d7aac4889f277c95b4f07b695dea90850d9f8dd13ad94a59f93e09289",
  "simHash": "9f1c...",
  "chunkCount": 6,
  "chunks": [
    {
      "chunkId": "198844c52f-2",
      "index": 2,
      "url": "https://www.joelonsoftware.com/2018/04/13/gamification/",
      "title": "A Dusting of Gamification – Joel on Software",
      "headingPath": ["A Dusting of Gamification"],
      "text": "We didn't want to punish you so much as we wanted to show other people that your answer was wrong...",
      "tokenCount": 424
    }
  ],
  "billedEvent": "page",
  "billedUnits": 1,
  "error": null
}
```

Linked documents are rows with `"type": "document"` plus `fileName`, `format` and `pageCount`. Free rows have `status` = `failed`, `blocked`, `robots-disallowed`, `duplicate`, `unchanged` or `skipped`, an empty `markdown` and a plain-language `error` / `duplicateOf`. Filter on `status = "ok"` to get the content.

### How much does it cost to crawl a website?

| Event | Price (Free / Bronze) | Silver | Gold | When |
|---|---|---|---|---|
| Page | $0.00015 | $0.000135 | $0.00013 | a page delivered over plain HTTP |
| Browser page | $0.0009 | $0.00081 | $0.00078 | a page that needed headless Chromium |
| Document | $0.002 | $0.0018 | $0.0017 | a linked PDF / DOCX / PPTX / XLSX converted |

Plus the standard Apify start fee ($0.00005 per GB of memory, $0.0001 at the default 2 GB). One event covers up to **25,000 tokens** of Markdown (about 18,000 words, longer than nearly every web page); a longer page or file counts one event per started 25,000 tokens. Failed, blocked, robots-disallowed, duplicate and unchanged pages cost **$0**.

#### Why a flat price beats usage billing (worked example)

You crawl a 1,000-page documentation site once a week.

- **Here:** 1,000 pages x $0.00015 = **$0.15 per crawl**, the same every week. Next week, with **Only changed pages**, the 950 pages that did not change are free: **about $0.01**.
- **Usage-billed crawlers** charge for compute time instead (memory x run time). Their own estimates for raw-HTTP crawling are around $0.2 per 1,000 pages, and $0.5 to $5 per 1,000 pages with a headless browser. The real bill moves with page weight, slow servers, retries and the memory you pick, so you only know it after the run.

Set **Max cost per run** in the run options and the crawler stops cleanly when the budget is reached.

### Use it with n8n, Make, LangChain and LlamaIndex

- **n8n / Make / Zapier:** run the Actor (or a saved task) with the Apify integration, then loop over the dataset items where `status` is `ok` and send `markdown` or each item of `chunks` to your vector store, Google Sheet or LLM step.
- **LangChain (Python):**

```python
from langchain_apify import ApifyWrapper
from langchain_core.documents import Document

loader = ApifyWrapper().call_actor(
    actor_id="kantolabs/website-content-crawler-ai",
    run_input={"startUrls": [{"url": "https://docs.example.com/"}], "maxPages": 200},
    dataset_mapping_function=lambda item: Document(
        page_content=item["markdown"], metadata={"source": item["url"], "title": item["title"]}),
)
docs = [d for d in loader.load() if d.page_content]
```

- **LlamaIndex:** use `ApifyActor` from `llama-index-readers-apify` with the same `actor_id` and map `item["markdown"]` (or each chunk's `text`) to a `Document`.
- **Plain API / agents:** `chunks.jsonl` from the OUTPUT record is one chunk per line with `url`, `headingPath`, `text` and `tokenCount`, ready to embed.

### Crawl scope, sitemaps and robots.txt

- **Scope:** by default only pages on the same host and under the start URL's folder are followed (`https://example.com/blog/` crawls `/blog/...`). Choose *Same host* or *Same domain and subdomains*, or set **Include URLs (globs)** such as `https://example.com/docs/**`.
- **Sitemaps:** sitemaps from robots.txt, `/sitemap.xml` and `/sitemap_index.xml` (including gzipped and nested ones) add in-scope pages that are not linked.
- **robots.txt:** respected by default for the `KantoCrawler` user agent. If robots.txt cannot be reached at all, the site is not crawled (RFC 9309). Turn **Respect robots.txt** off **only for a website you own or have written permission to crawl**.

### Limits

- Sites that block datacenter traffic or show a bot check return free `blocked` rows; this crawler does not use residential proxies, fingerprint tricks or CAPTCHA solving.
- No logins: pages behind a login are not crawled.
- Scanned PDFs without a text layer give a free error row (no OCR).
- Pages larger than **Max page size** (10 MB by default) are skipped with a free error row.

### Related Kanto Labs actors

- [Video to LLM Context](https://apify.com/kantolabs/video-to-llm-context) - keyframes per scene + transcript of a video, packaged for an LLM next to your website content.
- [Video OCR](https://apify.com/kantolabs/video-ocr) - on-screen text of video files (subtitles, captions, slides) as timed segments and SRT/VTT.
- [AI Image Caption & Alt Text Generator](https://apify.com/kantolabs/image-caption-generator) - captions, alt text and tags for images, e.g. the images of the pages you crawled.

### FAQ

#### Is it free to crawl a website?

You only pay for pages that are delivered with content: $0.15 per 1,000 plain-HTTP pages. Errors, blocked pages, duplicates and unchanged pages are free, and Apify's free plan credit covers thousands of pages per month.

#### Does it work on JavaScript websites (React, Vue, Angular)?

Yes. In *auto* mode the crawler fetches the plain HTML first and re-renders only the pages that are empty without JavaScript in headless Chromium (billed as Browser page). Choose *browser* to render every page.

#### How do I keep my vector database up to date?

Schedule the crawl and set **Only changed pages since this dataset** to the previous run's dataset. Unchanged pages come back free and marked `unchanged`, changed and new pages come back in full, and the OUTPUT record lists `possiblyRemovedUrls`.

#### What is the chunk size in?

Tokens of OpenAI's `cl100k_base` tokenizer (used by `text-embedding-3-*` and GPT-4-class models). Other tokenizers give similar counts for English.

#### Can I crawl many sites in one run?

Yes. Add several start URLs; each one keeps its own scope.

#### Something went wrong?

Every problem gets a clear `error` in its row and is free. If the Markdown of a page looks wrong, open an Issue with the URL and we will look at it.

# Actor input Schema

## `startUrls` (type: `array`):

Where the crawl starts, usually the top of a docs site, blog or help centre. Pages under the same path are followed (https://example.com/blog/ crawls /blog/...). A start URL that is a PDF/DOCX/PPTX/XLSX file is converted as a document.

## `crawlerType` (type: `string`):

auto (default): fast plain HTTP, and only pages that need JavaScript are re-rendered in headless Chromium (billed as Browser page). http: never use the browser (cheapest). browser: render every page in headless Chromium.

## `maxPages` (type: `integer`):

Most web pages fetched in this run (start URLs, linked and sitemap pages). Linked documents have their own limit.

## `maxCrawlDepth` (type: `integer`):

How many links away from a start URL to go. 0 = only the start URLs; sitemap pages count as depth 1.

## `crawlScope` (type: `string`):

Which links to follow when no include globs are set. startUrlPath (default): same host and under the start URL's folder. sameDomain: the whole host. sameDomainAndSubdomains: also docs./blog./www. subdomains.

## `includeUrlGlobs` (type: `array`):

If set, only links matching one of these globs are followed (replaces the crawl scope). \*\* = anything, \* = anything except /, {a,b} = either. Example: https://example.com/docs/\*\*. Write ? for a literal ?.

## `excludeUrlGlobs` (type: `array`):

Links matching any of these globs are never crawled (start URLs always are). Example: https://example.com/**/changelog/** or https://example.com/\*\*?page=\*.

## `useSitemaps` (type: `boolean`):

Read the site's sitemaps (from robots.txt, /sitemap.xml, /sitemap_index.xml) to find in-scope pages that are not linked.

## `respectRobotsTxt` (type: `boolean`):

On by default: pages that robots.txt disallows for crawlers are skipped, and Crawl-delay is honoured (max 10 s). Turn off ONLY for a website you own or have written permission to crawl.

## `chunkSize` (type: `integer`):

Split each page's Markdown into heading-aware chunks of at most this many tokens (OpenAI cl100k_base), each with URL and heading path. 0 = no chunks.

## `chunkOverlap` (type: `integer`):

Tokens of the previous chunk repeated at the start of the next one (cut at a sentence boundary when possible). At most half the chunk size.

## `convertLinkedDocuments` (type: `boolean`):

Also download PDF, Word, PowerPoint and Excel files linked from crawled pages (on any host, exclude globs still apply) and convert them to Markdown + chunks. Billed per document. Scanned PDFs without a text layer give a free error row (no OCR).

## `maxDocuments` (type: `integer`):

Most linked documents converted in this run.

## `maxDocumentSizeMb` (type: `integer`):

Larger linked files are skipped with a free error row.

## `previousDatasetId` (type: `string`):

Pick the dataset of an earlier run of this crawler. Pages whose content hash is the same (or that answer 304 Not Modified) become free 'unchanged' rows; you pay only for new and changed pages.

## `removeDuplicates` (type: `boolean`):

Pages with the same canonical URL, identical text or near-identical text (SimHash, e.g. print views, tag/sort variants) become free 'duplicate' rows pointing to the first copy.

## `saveLlmsTxt` (type: `boolean`):

Write llms.txt (page index), llms-full.txt (every page's Markdown in one file) and chunks.jsonl (all chunks, one JSON per line) to the run's key-value store. Links are in the OUTPUT record.

## `saveMarkdown` (type: `boolean`):

Put the page's main content as Markdown in each row.

## `saveText` (type: `boolean`):

Put the main content as plain text (no Markdown syntax) in each row.

## `keepLinks` (type: `boolean`):

Keep [text](url) links in the Markdown. Turn off for leaner LLM input (link text is kept).

## `keepImages` (type: `boolean`):

Keep ![alt](url) image references in the Markdown.

## `removeElementsCssSelector` (type: `string`):

Optional. Elements removed on top of the built-in cleaning (navigation, headers, footers, sidebars, cookie banners, ads, share widgets). Example: .promo, #newsletter-box

## `contentCssSelector` (type: `string`):

Optional. Keep only the elements matching this selector (e.g. article.post-body). Empty = automatic main-content detection.

## `maxConcurrency` (type: `integer`):

Pages fetched in parallel. Browser pages run fewer at a time (about one per 700 MB of memory).

## `requestTimeoutSecs` (type: `integer`):

Per-request timeout.

## `maxRetries` (type: `integer`):

Retries on network errors, 5xx and 429 answers. Blocked pages (bot checks, 401/403) are not retried.

## `browserWaitSecs` (type: `integer`):

Headless browser only: how long to wait for network quiet after the page loads.

## `maxPageSizeMb` (type: `integer`):

HTML pages larger than this are skipped with a free error row.

## `acceptLanguage` (type: `string`):

Language the crawler asks for on multilingual sites, e.g. de-DE,de;q=0.9.

## `proxyConfiguration` (type: `object`):

Optional. No proxy by default. Apify datacenter proxy can be used; residential proxy groups are not used.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.python-httpx.org/"
    }
  ],
  "crawlerType": "auto",
  "maxPages": 10,
  "maxCrawlDepth": 20,
  "crawlScope": "startUrlPath",
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "useSitemaps": true,
  "respectRobotsTxt": true,
  "chunkSize": 512,
  "chunkOverlap": 64,
  "convertLinkedDocuments": false,
  "maxDocuments": 50,
  "maxDocumentSizeMb": 25,
  "removeDuplicates": true,
  "saveLlmsTxt": true,
  "saveMarkdown": true,
  "saveText": true,
  "keepLinks": true,
  "keepImages": false,
  "maxConcurrency": 10,
  "requestTimeoutSecs": 30,
  "maxRetries": 2,
  "browserWaitSecs": 3,
  "maxPageSizeMb": 10,
  "acceptLanguage": "en-US,en;q=0.8,*;q=0.5",
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `llmsTxt` (type: `string`):

No description

## `llmsFullTxt` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.python-httpx.org/"
        }
    ],
    "maxPages": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("kantolabs/website-content-crawler-ai").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://www.python-httpx.org/" }],
    "maxPages": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("kantolabs/website-content-crawler-ai").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.python-httpx.org/"
    }
  ],
  "maxPages": 10
}' |
apify call kantolabs/website-content-crawler-ai --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,kantolabs/website-content-crawler-ai"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/R7cEk5LEyElWq2R9g/builds/wm6n4GWBewOsDllA3/openapi.json
