# Website to Markdown Crawler for AI, RAG & LLMs (`precious_bathmat/website-to-markdown`) Actor

Crawl any website or docs site and get every page as clean Markdown for ChatGPT, Claude, RAG and vector databases. Main content only, ready-made chunks, llms.txt and one combined file. Uses the sitemap, respects robots.txt. No browser, no API key.

- **URL**: https://apify.com/precious_bathmat/website-to-markdown.md
- **Developed by:** [Mariam Ahmed](https://apify.com/precious_bathmat) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.50 / 1,000 page as markdowns

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown Crawler for AI, RAG & LLMs

Crawl any **website, docs site or blog** and get every page as **clean Markdown**: the article or documentation text only, without menus, sidebars, footers, cookie banners or scripts. Ready for **ChatGPT, Claude, custom GPTs, RAG pipelines and vector databases**.

- **Main content only**: Mozilla's Readability finds the real text; headings, lists, tables, code blocks and links stay intact
- **Ready-made chunks for RAG**: split at headings and paragraphs to the token size you choose, with overlap and the heading path of every chunk
- **One combined Markdown file** for the whole site and an **llms.txt** index, ready to upload as a knowledge file
- **Finds pages through the sitemap and the links**, stays inside the section you give it, skips duplicates
- **Respects robots.txt**, include and exclude patterns, depth and page limits
- **No browser**: fast and cheap, about 2 pages per second on default memory

### What does this website to Markdown crawler do?

```
https://docs.python.org/3/tutorial/   → 17 pages (the whole tutorial)
https://react.dev/learn               → 51 pages
https://docs.apify.com/academy        → 200 pages
468 pages · 790,694 words · ~1.3 million tokens · 3,808 chunks of ~500 tokens
```

1. Give it **start addresses**: a whole site, a section (`/docs/`) or single pages.
2. It reads the **sitemap** and follows **links** inside that section, closest pages first, until your page limit.
3. Each page becomes **Markdown** with its title, description, word and token count, and **chunks** if you ask for them.

```mermaid
flowchart LR
    A[Start address] --> B[Sitemap + links<br/>inside the section]
    B --> C[Main content<br/>via Readability]
    C --> D[Markdown per page<br/>+ RAG chunks]
    D --> E[ALL_MARKDOWN file<br/>+ llms.txt]
```

### Who is it for?

- **AI developers** feeding documentation into RAG, embeddings and vector databases (Pinecone, Qdrant, Weaviate, pgvector)
- **Teams building custom GPTs and Claude projects** that need a site's content as one knowledge file
- **Support teams** turning a help center into the source for an AI assistant
- **Researchers and writers** archiving blogs and essays as clean text
- **Developers** migrating a website's content into Markdown for a static site or a new CMS

### What data do you get?

**Pages** (dataset, one row per page)

| Field | What it tells you |
|---|---|
| `url`, `title`, `description`, `h1`, `canonicalUrl`, `image` | The page |
| `markdown` | The page's text as Markdown, starting with its heading |
| `wordCount`, `tokens` | Size; tokens are estimated at 4 characters each |
| `chunks` | When **Chunk size** is set: `index`, `text`, `headings` (where in the page it sits), `tokens`, `url`, `title` |
| `language`, `author` | From the page |
| `extractedFrom` | Main content, or the whole page when the page has no clear article |
| `depth`, `foundVia`, `startUrl`, `crawledAt` | How the crawler reached it |

The key-value store also has **`ALL_MARKDOWN`** (every page in one Markdown file, each starting with its source address), **`LLMS_TXT`** (an [llms.txt](https://llmstxt.org) index of the pages), **`PAGES_CSV`** and **`RUN_SUMMARY`** (what was skipped and why).

A chunk from `react.dev/learn/thinking-in-react`:

```json
{ "index": 2, "headings": ["Thinking in React", "Step 1: Break the UI into a component hierarchy"], "tokens": 239,
  "text": "displays and filters the list according to the user input.\n4.  `ProductCategoryRow` (green) displays a heading..." }
```

### Example: 8 October 2026

**4 start addresses, 468 pages in 3 minutes 38 seconds**, with 500-token chunks.

| Start address | Pages | How it found them |
|---|---|---|
| docs.python.org/3/tutorial/ | 17, the whole tutorial | links |
| react.dev/learn | 51, the whole Learn section | links |
| docs.apify.com/academy | 200 (the limit) | sitemap and links |
| paulgraham.com/articles.html | 200 (the limit) | links |

- **790,694 words, about 1.3 million tokens, 3,808 chunks.**
- 461 pages had a clear main text; 7 index pages were kept whole.
- Skipped without charge: 35 links to files that are not pages (PDF, images), 1 duplicate page, and everything outside the four sections.

### Pricing

**$0.0005 per page** saved as Markdown. Skipped pages, duplicates and pages that cannot be read are free.

- 1,000 pages: **$0.50**
- The run above: 468 pages, **$0.23**

### Good to know

- **JavaScript-only sites**: pages that build all their text in the browser have no text in their HTML; they are listed in `RUN_SUMMARY` and not charged. Most docs sites, blogs and marketing sites serve their text in the HTML.
- **Stay in a section**: by default the crawler stays under the start address (`/docs/` crawls `/docs/...`). Choose **Whole website** to follow every link on the site, or **Only these pages** to read just the listed addresses.
- **Sites that block bots** with a challenge page are listed in `RUN_SUMMARY`.
- **robots.txt** is respected by default; the start addresses you list are always read.
- **Big crawls**: thousands of pages work; for more speed give the run more memory. `ALL_MARKDOWN` holds up to 50 MB; the dataset always has every page.
- No login, no API key, no proxies needed.

### Input

| Field | Meaning |
|---|---|
| **Websites or pages** | Start addresses, one per line |
| **Maximum pages per website** | 1 to 10,000 (default 50) |
| **What to crawl** | Section under the start address (default), whole website, or only these pages |
| **Content** | Main content (default) or whole page |
| **Chunk size for RAG**, **Chunk overlap** | Tokens; 0 = no chunks (default); overlap 50 |
| **Only pages matching**, **Skip pages matching** | Patterns like `/blog/**` |
| **Maximum link depth** | Default 20 |
| **Use the sitemap**, **Respect robots.txt** | Both on by default |
| **Keep links**, **Keep images** | Links on, images off |
| **Skip duplicate pages** | On by default |
| **Pages at once** | 1 to 20 (default 5) |

### Integrations

Export to JSON, CSV or Excel, or connect through the Apify API, webhooks, Make, Zapier, n8n and LangChain. Upload `ALL_MARKDOWN` to a custom GPT or a Claude project, or send the `chunks` straight to your embedding model.

# Actor input Schema

## `startUrls` (type: `array`):

One per line: a site (example.com), a section (https://docs.apify.com/platform) or single pages. Crawling starts here.

## `maxPagesPerSite` (type: `integer`):

Stops after this many pages on each website. Pages closest to the start address come first.

## `crawlScope` (type: `string`):

"Section" stays under the start address (https://site.com/docs/ crawls /docs/... only). "Whole website" follows every link on the same site. "Only these pages" reads the listed addresses and nothing else.

## `contentMode` (type: `string`):

"Main content" keeps the article or documentation text and drops menus, sidebars, footers and cookie banners. "Whole page" keeps every visible text block except menus, footers, scripts and banners.

## `chunkTokens` (type: `integer`):

0 = no chunks. Otherwise each page also gets a "chunks" list cut at headings and paragraphs, about this many tokens each, with the heading path of every chunk. 300-800 suits most embedding models.

## `chunkOverlapTokens` (type: `integer`):

How much of the previous chunk is repeated at the start of the next one, so no sentence loses its context.

## `includeUrlGlobs` (type: `array`):

Optional patterns. Pages not matching any of them are skipped. A path pattern like "/blog/\*\*" or a full one like "https://example.com/docs/\*". \* = anything but /, \*\* = anything.

## `excludeUrlGlobs` (type: `array`):

Optional patterns for pages to leave out, e.g. "/tag/**", "/page/\*", "**?lang=\*".

## `maxDepth` (type: `integer`):

How many clicks away from the start address to go. Pages found in the sitemap count as depth 1.

## `useSitemap` (type: `boolean`):

Also read the site's sitemap.xml to find pages no link points to. Finds most pages of a docs site in one request.

## `respectRobotsTxt` (type: `boolean`):

Skip pages the site's robots.txt asks crawlers not to read. Start addresses you list are always read.

## `keepLinks` (type: `boolean`):

Keep links as [text](address). Off gives plain text, which uses fewer tokens.

## `keepImages` (type: `boolean`):

Keep images as ![alt](address).

## `skipDuplicateContent` (type: `boolean`):

Save a page once when several addresses show exactly the same text (print views, tracking links, language redirects).

## `concurrency` (type: `integer`):

How many pages are read in parallel. Lower it for small sites that slow down.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com/academy/web-scraping-for-beginners"
  ],
  "maxPagesPerSite": 50,
  "crawlScope": "path",
  "contentMode": "main",
  "chunkTokens": 0,
  "chunkOverlapTokens": 50,
  "maxDepth": 20,
  "useSitemap": true,
  "respectRobotsTxt": true,
  "keepLinks": true,
  "keepImages": false,
  "skipDuplicateContent": true,
  "concurrency": 5
}
```

# Actor output Schema

## `pages` (type: `string`):

One row per page: Markdown, title, description, words, tokens and chunks when asked for.

## `allMarkdown` (type: `string`):

Every page one after another, each starting with its source address. Paste into ChatGPT or Claude, or upload as a knowledge file.

## `llmsTxt` (type: `string`):

An llms.txt index of the crawled pages: title, address and description.

## `summary` (type: `string`):

Pages, words and tokens, what was skipped and why.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com/academy/web-scraping-for-beginners"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("precious_bathmat/website-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://docs.apify.com/academy/web-scraping-for-beginners"] }

# Run the Actor and wait for it to finish
run = client.actor("precious_bathmat/website-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com/academy/web-scraping-for-beginners"
  ]
}' |
apify call precious_bathmat/website-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,precious_bathmat/website-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/O1XCvnFpFjKcdy7Am/builds/k1PequK1Wu1q1r9fT/openapi.json