# Website to Markdown Crawler for RAG & llms.txt — Docsieve (`cybermax/website-to-markdown`) Actor

Website to Markdown crawler for RAG and LLMs: crawl any website or docs site and get clean Markdown per page (no menus or footers), RAG-ready chunks with token counts, and llms.txt + llms-full.txt. Monitor mode returns only changed pages. $1 per 1,000 pages.

- **URL**: https://apify.com/cybermax/website-to-markdown.md
- **Developed by:** [CyberMax](https://apify.com/cybermax) (community)
- **Categories:** AI, Developer tools, Open source
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.00 / 1,000 page converteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Docsieve: Website to Markdown Crawler for RAG, LLMs & llms.txt

#### Turn any website or docs site into clean Markdown, RAG-ready chunks and llms.txt, and keep it fresh

Point Docsieve at a website or documentation site. It crawls it like a search engine (sitemap + internal links, robots.txt respected), strips menus, headers, footers, cookie banners and scripts, and converts each page's main content to clean Markdown with code fences (language kept), tables and absolute links. Every page comes with heading-aware chunks and token estimates, and the run also writes **llms.txt** and **llms-full.txt** for the site. On a schedule, monitor mode returns only new and changed pages, so your vector index stays current for pennies. **$1 per 1,000 pages, no start fee.**

**[▶ Try it now with a prefilled input](https://apify.com/cybermax/website-to-markdown)** · $0.001 per page (a 500-page docs site = $0.50) · llms.txt files free · Apify's free plan includes $5 of credit a month

**Built for:** developers building RAG chatbots and AI assistants on their own docs · AI agents that need a site's content as text · teams publishing llms.txt for AI search (GEO) · knowledge-base and support-bot builders · researchers collecting web text

**Real output** (real run on Apify, 10 Nov 2026: Apify Academy, `"includePaths": ["/academy*"]`, 10 pages in 5.2 s, platform cost $0.0001; four rows shown):

| Page | Words | Tokens (est.) | Chunks |
|---|---|---|---|
| /academy | 254 | 489 | 1 |
| /academy/advanced-web-scraping/crawling/sitemaps-vs-search | 1,000 | 1,581 | 4 |
| /academy/ai/ai-agents | 1,321 | 2,540 | 6 |
| /academy/anti-scraping | 2,551 | 4,271 | 11 |

Start of the first page's Markdown:

```markdown
## Apify Academy

Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.

### Beginner courses

##### Web scraping with AI

Learn how to use AI to extract information from websites in this practical course, starting from the absolute basics.

<https://docs.apify.com/academy/scraping-with-ai>
```

```json
{"startUrls": ["https://docs.apify.com/academy"], "includePaths": ["/academy*"], "maxPages": 10}
```

### Ready-made inputs (one click)

| Job | Input | Result |
|---|---|---|
| **Docs → RAG knowledge base** | `{"startUrls": ["https://docs.yourproduct.com"], "maxPages": 2000, "outputMode": "chunks"}` | One row per chunk, ready to embed. |
| **Generate llms.txt** | `{"startUrls": ["https://yoursite.com"], "maxPages": 300, "chunking": false}` | `LLMS-TXT` and `LLMS-FULL-TXT` in the run's storage. |
| **Just the blog** | `{"startUrls": ["https://site.com/blog"], "includePaths": ["/blog/*"], "excludePaths": ["/blog/tag/*"]}` | Every post as Markdown. |
| **Keep a chatbot fresh** | `{"startUrls": ["https://docs.yourproduct.com"], "monitor": true, "outputMode": "chunks"}` | On a schedule: only new and changed pages; removed pages listed. |
| **Several sites at once** | `{"startUrls": ["https://a.com", "https://b.com"], "maxPages": 100}` | A summary and llms.txt per site. |

### Output

**Pages mode (default):** one row per page with `url`, `title`, `description`, `lang`, `canonical`, `depth`, `markdown`, `wordCount`, `tokens`, `contentHash`, `crawledAt`, `chunkCount` and `chunks[]` (`index`, `heading`, `headings`, `text`, `chars`, `tokens`).

**Chunks mode:** one row per chunk with the page fields plus `chunkId` (stable: content hash + index), `chunkIndex`, `chunkCount`, `heading` (path such as `Guide > Install > Linux`), `text` and `chunkTokens`. Upsert by `chunkId` into Pinecone, Qdrant, Weaviate, pgvector, Chroma or Milvus; LangChain and LlamaIndex read Apify datasets directly.

**Key-value store:** `LLMS-TXT` (llms.txt: site name, summary and a link list grouped by section), `LLMS-FULL-TXT` (every page's Markdown with its source URL) and `OUTPUT` (per site: pages converted, total tokens, pages skipped by robots.txt / noindex / thin content, failures, and in monitor mode the changed and removed URLs).

### Run it on a schedule (fresh RAG index)

1. Save your input with `"monitor": true` as a **task**.
2. **Schedules → Create**: daily or weekly (cron `0 3 * * *`).
3. Add a webhook "when the run succeeds" to your ingestion job (or a Make/n8n flow) that upserts the new chunks and deletes chunks of `removedSinceLastRun` pages. Unchanged pages cost nothing.

### Integrations

- **LangChain (Python):** `ApifyDatasetLoader(dataset_id=..., dataset_mapping_function=lambda d: Document(page_content=d["text"], metadata={"source": d["url"], "heading": d["heading"]}))`
- **LlamaIndex:** the Apify reader with the same mapping.
- **API (curl):**
  ```bash
  curl -X POST "https://api.apify.com/v2/acts/cybermax~website-to-markdown/run-sync-get-dataset-items?token=YOUR_TOKEN" \
    -H "Content-Type: application/json" -d '{"startUrls": ["https://docs.apify.com/academy"], "maxPages": 10}'
  ```
- **Make / Zapier / n8n:** the Apify app or node, then your vector database's node.
- **AI agents (MCP):** add `cybermax/website-to-markdown` to the Apify MCP server (`https://mcp.apify.com?actors=cybermax/website-to-markdown`) so an agent can read a whole site as Markdown.

### Why Docsieve

| | Docsieve | Typical website-to-Markdown Actor on the Store |
|---|---|---|
| Price | $1 per 1,000 pages, no start fee | Most-used pay-per-page rivals: $1.56 to $3 per 1,000 pages, some with a start fee; range on the Store $0.20 to $30 (Store prices on 10 Nov 2026) |
| Chunks | Heading-aware, size and overlap you choose, token estimates, chunk-per-row mode | Often whole pages only |
| llms.txt | llms.txt + llms-full.txt built every run, free | Separate tools |
| Re-runs | Monitor mode: only changed pages, removed pages listed | Full re-crawl every time |
| Clean text | Main content only; code language, tables and absolute links kept | Varies |

### FAQ

**Does it render JavaScript?** No: it reads the HTML the server sends, which keeps it fast and cheap. That covers most docs sites, blogs and marketing sites (Docusaurus, GitBook, MkDocs, Mintlify, WordPress, Next.js with server rendering). Pages that only exist after client-side rendering come back thin and are skipped (and not charged).

**How are tokens counted?** Estimated at 4 characters per token, close to GPT and Claude tokenizers for English text.

**Is it legal?** Docsieve reads public pages and obeys robots.txt; it skips pages marked noindex unless you ask for them. Convert sites you own or have the right to use, and check each site's terms for how you may use its content.

**Why not Apify's Website Content Crawler?** It is excellent and renders JavaScript, but you pay its compute for every page on every run. Docsieve is a fixed $0.001 per page, builds chunks and llms.txt for you, and on scheduled runs charges only for pages that changed.

### Also from CyberMax

- [Inkpluck: Article Extractor](https://apify.com/cybermax/article-extract) · single articles or RSS feeds to clean text and Markdown.
- [Sitemoss: Sitemap URL Extractor](https://apify.com/cybermax/sitemap-url-extractor) · every URL a site lists.
- [Auditmoss: SEO Audit Crawler](https://apify.com/cybermax/seo-audit) · SEO issues and broken links across a site.

Found this useful? A quick rating helps others find it.

© 2026 CyberMax. All rights reserved.

# Actor input Schema

## `startUrls` (type: `array`):

Start page of each site, e.g. https://docs.example.com. Pages are found from the sitemap and by following internal links.

## `maxPages` (type: `integer`):

Pages converted (and charged) per site.

## `includePaths` (type: `array`):

Optional, \* = anything. E.g. /docs/\* or /blog/\*. The start page is always read.

## `excludePaths` (type: `array`):

Optional, e.g. /tag/*, /author/*, /changelog/\*.

## `maxDepth` (type: `integer`):

Clicks from the start page to follow.

## `outputMode` (type: `string`):

Chunk rows carry the page URL, title and heading path, so you can load them straight into Pinecone, Qdrant, Weaviate, pgvector or Chroma. Charged per page either way.

## `chunking` (type: `boolean`):

Heading-aware chunks with token estimates.

## `chunkSize` (type: `integer`):

About 500 tokens at the default.

## `chunkOverlap` (type: `integer`):

Text repeated from the previous chunk for context.

## `llmsTxt` (type: `boolean`):

Saved in the run's key-value store (free).

## `includeLinks` (type: `boolean`):

Keep [text](url) links in the Markdown.

## `includeImages` (type: `boolean`):

Keep ![alt](src) images in the Markdown.

## `minWords` (type: `integer`):

Skips empty and redirect-only pages (not charged).

## `useSitemap` (type: `boolean`):

Seed the crawl from robots.txt sitemaps or /sitemap.xml.

## `includeSubdomains` (type: `boolean`):

Also crawl docs.example.com when starting at example.com.

## `includeNoindex` (type: `boolean`):

Pages marked noindex are skipped unless this is on.

## `maxConcurrency` (type: `integer`):

Lower it for small servers.

## `monitor` (type: `boolean`):

For scheduled runs. Returns only new and changed pages (by content hash) and lists removed pages in the summary. Unchanged pages are free.

## `monitorKey` (type: `string`):

Optional name for this crawl's saved state.

## `proxyConfiguration` (type: `object`):

Optional. Only needed if a site blocks or rate-limits you.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "maxPages": 10,
  "includePaths": [
    "/academy*"
  ],
  "maxDepth": 10,
  "outputMode": "pages",
  "chunking": true,
  "chunkSize": 2000,
  "chunkOverlap": 200,
  "llmsTxt": true,
  "includeLinks": true,
  "includeImages": false,
  "minWords": 30,
  "useSitemap": true,
  "includeSubdomains": false,
  "includeNoindex": false,
  "maxConcurrency": 5,
  "monitor": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `llmsTxt` (type: `string`):

No description

## `llmsFullTxt` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com/academy"
    ],
    "maxPages": 10,
    "includePaths": [
        "/academy*"
    ],
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("cybermax/website-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.apify.com/academy"],
    "maxPages": 10,
    "includePaths": ["/academy*"],
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("cybermax/website-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "maxPages": 10,
  "includePaths": [
    "/academy*"
  ],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call cybermax/website-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,cybermax/website-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/lrd5nNu4L0D46ZWpc/builds/DSaoGs1Bcv0Cjkcs7/openapi.json