# Docs to Markdown for AI Chatbots & RAG (`hustle_staxx/docs-markdown-crawler`) Actor

Turn any documentation site into clean Markdown chunks, split at headings and ready for RAG, vector databases and AI agents. Turn on incremental mode and re-runs deliver only new or changed pages, so keeping your knowledge base fresh costs a fraction. Obeys robots.txt.

- **URL**: https://apify.com/hustle_staxx/docs-markdown-crawler.md
- **Developed by:** [Hustle Staxx](https://apify.com/hustle_staxx) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 page delivereds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Docs to LLM Markdown (incremental)

Turns a documentation or help-centre site into clean Markdown, split into heading-aware chunks ready for a RAG pipeline or vector database. Turn on **incremental mode** for scheduled runs and it only delivers pages that changed since the last run, so you pay the full page price for, and re-embed, only what is new.

**Incremental vs. full re-crawls:** a full re-crawl delivers and charges every page on every run. In incremental mode an unchanged page costs a fifth of a delivered page and is not delivered again. For a 200-page site where 20 pages change a day, a daily full re-crawl costs $0.22 a day; incremental mode costs about $0.08 a day after the first run (worked out under Pricing).

### What you get

One dataset item per page:

| Field | Meaning |
| --- | --- |
| `url`, `title` | The page |
| `status` | `new` or `changed` since your last run |
| `contentHash` | SHA-256 of the page Markdown |
| `tokenEstimate` | Rough token count (about 4 characters per token) |
| `markdown` | The whole page as Markdown |
| `chunks` | `index`, `headings` (the heading path, such as `["Install", "With pip"]`), `markdown`, `tokenEstimate` |
| `crawledAt` | UTC time of the fetch |

Navigation, headers, footers, scripts and forms are removed, including menus, sidebars, tables of contents and breadcrumbs that a site builds from plain `div`s (recognised by their class, id or role, and only when they are mostly links and hold no main content). Code blocks keep their language, and tables become Markdown tables.

### Example input

```json
{
  "startUrl": "https://docs.example.com/guide/",
  "maxPages": 200,
  "incremental": true,
  "delaySeconds": 1,
  "maxChunkChars": 4000
}
```

Only `startUrl` matters for a first try; the rest are the defaults, except `maxPages` (default 50) and `incremental` (default off).

### Example output item

One page from an example site (the site and text are made up for illustration):

````json
{
  "url": "https://docs.example.com/guide/install",
  "title": "Install | Widget docs",
  "status": "changed",
  "contentHash": "51a0b14b5e2192b8d080165a70db11ee2d2b2cf396da4afec38d9ce946177505",
  "tokenEstimate": 34,
  "crawledAt": "2026-10-15T06:00:12+00:00",
  "markdown": "# Install\n\nWidget needs **Python 3.11** or later.\n\n## With pip\n\n```bash\npip install widget\n```\n\nCheck it worked with `widget --version`.",
  "chunks": [
    {
      "index": 0,
      "headings": ["Install"],
      "markdown": "# Install\n\nWidget needs **Python 3.11** or later.",
      "tokenEstimate": 12
    },
    {
      "index": 1,
      "headings": ["Install", "With pip"],
      "markdown": "## With pip\n\n```bash\npip install widget\n```\n\nCheck it worked with `widget --version`.",
      "tokenEstimate": 21
    }
  ]
}
````

### How it crawls

- Reads `robots.txt` first and always obeys it, including `Crawl-delay`. This cannot be turned off.
- Uses the site's sitemaps first, then follows links. It stays on the same site and under the start URL's folder.
- Fetches one page at a time, with at least 0.5 seconds between requests, and stops at once if the site answers 429 or 503.
- Skips pages that need a login, pages marked `noindex`, and pages that look like people listings (team, staff or author pages, or pages with several email addresses). Any other email address or phone number is replaced with `[email removed]` or `[phone removed]`.
- Refuses IP addresses and hosts that point to private or internal networks, including on redirects.

### Your responsibility

Only crawl sites whose terms allow automated access and reuse of their content, such as your own docs or docs published under an open licence. The crawler takes public pages only and does not bypass logins or captchas.

### For site owners

This crawler identifies itself as `HustleStaxxDocsCrawler` and obeys robots.txt. To block it, add this to your robots.txt:

```
User-agent: HustleStaxxDocsCrawler
Disallow: /
```

### Pricing

You pay per event, in US dollars:

| Event | Price | When |
| --- | --- | --- |
| Run start | $0.02 | Once per run |
| Page delivered | $0.001 | Each new or changed page written to the dataset |
| Page unchanged | $0.0002 | Incremental mode only: each page checked and found the same as last run |

Skipped pages (blocked by robots.txt, needing a login, not HTML, and so on) cost nothing.

**Worked example: a 200-page docs site** (set `maxPages` to 200 or more)

- First run: $0.02 + 200 × $0.001 = **$0.22**.
- Each daily run after that in incremental mode, if 20 pages changed and 180 did not: $0.02 + 20 × $0.001 + 180 × $0.0002 = $0.02 + $0.02 + $0.036 = **about $0.08**.
- The same daily run as a full re-crawl: **$0.22** every day.
- Over 30 days: about $2.42 in incremental mode ($0.22 + 29 × $0.076) against $6.60 for daily full re-crawls.

Set a maximum cost per run in the Apify Console to cap spending; the crawler stops cleanly when it is reached and the run still ends as a success with the pages delivered so far.

### Limits

- One site per run: the same host as the start URL, and only pages under the start URL's folder.
- Up to 10,000 pages per run (`maxPages`, default 50). Pages that are skipped count toward the limit, because they were still fetched.
- One request at a time, at least 0.5 seconds apart (default 1 second, longer if robots.txt asks). A 200-page site takes at least 3 to 4 minutes.
- HTML pages only. PDFs and other files are skipped, and so are pages larger than 5 MB.
- It does not run JavaScript, log in or solve captchas.
- Reads at most 20 sitemap files.
- Chunk size is set by `maxChunkChars` (default 4,000 characters, minimum 500). Chunks split only between blocks, so a single code block, table or paragraph longer than the limit makes one chunk longer than the limit.

### Troubleshooting

- **The run failed with "Could not read any page: robots.txt does not allow the start URL".** The site's robots.txt blocks this crawler or all crawlers from that folder. The crawler always obeys robots.txt and cannot be made to ignore it. Ask the site owner, or crawl a site you own or one that allows it.
- **The run failed with "Could not read any page: ..." and another reason.** No page could be read, so the run is marked as failed instead of succeeding with an empty dataset. The message says why, for example `robots.txt returned 500` (the site is down; try later) or `every page was skipped (12 needs login or forbidden)` (the docs are behind a login). You still pay the $0.02 start fee for that run.
- **Pages come back empty or are skipped as "no text content".** The site is probably built with JavaScript only: the page text is drawn in the browser and is not in the HTML. This crawler does not run JavaScript. Look for a static version of the docs (many projects publish them on GitHub or as a plain HTML export) and use that as the start URL.
- **The run stopped early with "site asked us to slow down (429)".** Raise `delaySeconds` and run again.
- **Some pages are missing.** Check they sit under the start URL's folder, that they are linked from other pages or listed in the sitemap, and that `maxPages` is high enough. The status message lists every reason a page was skipped.
- **Incremental mode delivered every page again.** The first incremental run for a site has nothing to compare with, so every page is new. Changes are tracked per site and start folder, so a different start URL starts a fresh history.

### Change log

#### 0.2 (2026-10-15)

- A run that cannot read a single page (for example, blocked by robots.txt or the site is down) now fails with a clear message, instead of ending as a success with 0 pages. A run that stops at your spending limit still counts as a success.
- Menus, sidebars, tables of contents and breadcrumbs built from plain `div`s are now removed from the text.
- New tests: the page limit is never exceeded, div-built sidebars are removed without losing main content, and a run with nothing read fails.
- README: prices in money with a worked example, example input and output, limits, troubleshooting and this change log.

#### 0.1 updates (2026-10-14)

- Incremental mode is now off by default (turn it on for scheduled runs), and the start URL defaults to the Apify Academy docs so a run with empty input works.
- Hardened against hostile pages: email and phone masking stays fast on crafted text, a page nested too deeply to read is skipped instead of failing the run, and pages over 5 MB are skipped.
- Added output and dataset schemas, so the Output tab shows an overview table.

#### 0.1 (2026-10-14)

- First release.

### Local development

```bash
python -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt pytest
python -m pytest -q   # uses a fake site, no network needed
```

# Actor input Schema

## `startUrl` (type: `string`):

The docs home page, for example https://docs.example.com/guide/. Only pages under this folder on the same site are crawled.

## `maxPages` (type: `integer`):

Upper limit on pages fetched in one run.

## `incremental` (type: `boolean`):

Remembers each page's content hash between runs and only outputs new or changed pages. Turn on for scheduled runs that feed a RAG index; leave off to get every page.

## `delaySeconds` (type: `number`):

Polite pause between requests. A longer Crawl-delay in the site's robots.txt always wins.

## `maxChunkChars` (type: `integer`):

Chunks split at headings, and long sections split further at paragraph boundaries.

## Actor input object example

```json
{
  "startUrl": "https://docs.apify.com/academy/",
  "maxPages": 50,
  "incremental": false,
  "delaySeconds": 1,
  "maxChunkChars": 4000
}
```

# Actor output Schema

## `results` (type: `string`):

One record per page: URL, title, status, token estimate, full Markdown and heading-aware chunks.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrl": "https://docs.apify.com/academy/"
};

// Run the Actor and wait for it to finish
const run = await client.actor("hustle_staxx/docs-markdown-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrl": "https://docs.apify.com/academy/" }

# Run the Actor and wait for it to finish
run = client.actor("hustle_staxx/docs-markdown-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrl": "https://docs.apify.com/academy/"
}' |
apify call hustle_staxx/docs-markdown-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,hustle_staxx/docs-markdown-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/uhxRQkLj3BFn8TN1a/builds/82ufoSsRYxSlg4bNg/openapi.json