# Docs to RAG Crawler: Site to Markdown | Johto Data (`johto.data/site-markdown-crawler`) Actor

Crawl a docs site into token-budgeted Markdown for RAG. llms.txt-first. Re-crawl incrementally: pass last hashes and unchanged or duplicate pages are free. $2.50 per 1,000 crawled pages.

- **URL**: https://apify.com/johto.data/site-markdown-crawler.md
- **Developed by:** [Jo](https://apify.com/johto.data) (community)
- **Categories:** AI, Agents, MCP servers
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Docs to RAG Crawler: Site to Markdown

By Johto Data. Crawl a docs site or blog into token-budgeted Markdown, one row per page. Re-crawls charge only new and changed pages.

### For AI agents

- **Call:** run `johto.data/site-markdown-crawler` with the example input below and read the default dataset.
- **Re-crawls:** store each row's `url` and `stats.contentHash`, pass them back as `"knownHashes": {"<url>": "<hash>"}`. Unchanged and duplicate pages are skipped and not charged; their links are still followed.
- **Context size:** `stats.tokensEstimate` per page; `maxTokens` trims long pages (`stats.truncated` flags it).
- **Scope:** URLs under a start URL, unless you set `includeUrlGlobs` / `excludeUrlGlobs`. Cap spend with `maxCrawlPages` or `maxResults`.
- **Single-page mode:** `"mode": "single-page"` with `urls` (or `urls` and no `startUrls`). One row per URL, same fields as llm-web-reader.
- **Safety:** http(s) only. Each host is resolved once and refused unless every address is a routable internet address (private, loopback, link-local and metadata addresses are refused); the connection goes only to that checked address, and every redirect hop is re-checked. Refused URLs return a status row at no charge. robots.txt is always respected.

| | This Actor | Leading alternative |
|---|---|---|
| Crawled page | $0.00025 | $0.0005 |
| Single page | $0.0002 (60% under) | $0.0005 |
| Run start | $0.00005 | none |
| Measured run time | 10-page crawl: 5.3 s; 1 single page: 4.2 s | not measured |

Sample (run mGSasox4jPWgK5bGS, Oct 22 2026, input below):

| url | metadata.title | stats.tokensEstimate |
|---|---|---|
| https://docs.python.org/3/tutorial/ | The Python Tutorial | 703 |
| https://docs.python.org/3/tutorial/appetite.html | 1. Whetting Your Appetite | 1120 |
| https://docs.python.org/3/tutorial/modules.html | 6. Modules | 6384 |

### Example input (crawls 10 pages)

```json
{
  "startUrls": [{"url": "https://docs.python.org/3/tutorial/"}],
  "maxCrawlPages": 10,
  "maxCrawlDepth": 1
}
```

Cost: $0.00255 (10 × $0.00025 + $0.00005).

### Output schema

One row per saved page:

- `url`: final page URL
- `crawl`: `loadedUrl`, `loadedTime`, `referrerUrl`, `depth`, `httpStatusCode`
- `metadata`: `canonicalUrl`, `title`, `description`, `author`, `keywords`, `languageCode`, `publishedAt`, `url`
- `text`, `markdown`; `html` only when `saveHtml` is true
- `stats`: `markdownChars`, `tokensEstimate` (chars/4), `tokensEstimateFull`, `truncated`, `contentHash`

The run's `OUTPUT` record holds `savedPages`, `unchangedPagesSkipped`, `duplicatePagesSkipped`, `chargedPages`.

### Single-page mode

```json
{"mode": "single-page", "urls": ["https://docs.python.org/3/tutorial/appetite.html"], "outputFormats": ["markdown"], "maxTokens": 1500}
```

Per URL: `crawl` (status, loadedUrl, loadedAt, requestStatus, `error` on failure), `searchResult`, `metadata`, `markdown` / `text` / `html` per `outputFormats`, `stats`, `links` (if `includeLinks`), `unchanged`. Failed, robots-blocked and unchanged rows are not charged. Keyword search is not supported: pass page URLs.

### Pricing

- `apify-actor-start`: $0.00005 per run
- `page`: $0.00025 per saved page (crawl mode)
- `single-page`: $0.0002 per page read (single-page mode)

### Use it when / not when

- Use it for docs sites and blogs that render as plain HTML, for RAG indexes, embeddings or LLM context, and for scheduled re-crawls that should bill only changes.
- Do not use it for pages that need JavaScript, files, screenshots, logins, or for structured records such as prices. Use apify/website-content-crawler for JavaScript sites.

### Compatibility

Input `startUrls`, `maxCrawlPages`, `maxCrawlDepth`, `includeUrlGlobs`, `excludeUrlGlobs`, `useSitemaps`, `useLlmsTxt`, `saveMarkdown`, `saveHtml`, `htmlTransformer`, `removeElementsCssSelector`, `maxResults` and output `url`, `crawl`, `metadata`, `text`, `markdown` use the same names as Apify's Website Content Crawler.

- **MCP:** `https://mcp.apify.com/?tools=johto.data/site-markdown-crawler`
- **LangChain / LlamaIndex:** `ApifyActorsTool("johto.data/site-markdown-crawler")`

### Limits

Raw HTTP only (no JavaScript rendering). Token counts are estimates. You are responsible for the URLs you supply and for following each site's terms.

Made by [Johto Data](https://johtodata.com).

# Actor input Schema

## `mode` (type: `string`):

crawl = follow links from startUrls. single-page = read exactly the URLs in `urls` or `query`, one row each, with the same output fields as llm-web-reader. Single-page mode is also used automatically when `urls` or `query` is set and `startUrls` is empty.

## `query` (type: `string`):

The web page URL to read, e.g. `https://docs.apify.com/platform/actors`. Same field name as apify/rag-web-browser. Several URLs separated by spaces are fetched in parallel; commas are kept as part of the URL. Keyword search is NOT supported: search with your own tool, then pass the URLs here. Use when: you have one page URL (or a few, space-separated). Don't use when: you only have keywords (search first) or the page needs JavaScript to render. You are responsible for the URLs you supply and for following the target site's terms.

## `urls` (type: `array`):

Optional list of page URLs to read in the same call (batch). Each item is one URL, used as given. Each successful page is one `single-page` event ($0.0002 per page, 60% under the $0.0005 per page that same-output rivals charge). Use when: you read a batch of URLs in one run (one start fee per batch). You are responsible for the URLs you supply and for following the target site's terms.

## `outputFormats` (type: `array`):

Formats to return for each page. `markdown` (default) is best for LLMs. Same values as apify/rag-web-browser.

## `includeLinks` (type: `boolean`):

Also return the page's absolute links (up to 500) in `links`, for agents that browse onward. Use when: your agent browses onward from this page.

## `maxUrls` (type: `integer`):

Safety cap on how many URLs one run reads in single-page mode. Use it to cap spend: each page read costs $0.0002 (event `single-page`).

## `startUrls` (type: `array`):

Where to start, e.g. `[{"url": "https://docs.apify.com/academy"}]`. Only pages whose URL starts with a start URL are crawled (same rule as apify/website-content-crawler), unless you set includeUrlGlobs. Use when: you want a whole docs site, help center or blog. Don't use when: you have one known page (use johto.data/llm-web-reader) or the site needs JavaScript to render. You are responsible for the URLs you supply and for following the target site's terms.

## `maxCrawlPages` (type: `integer`):

Maximum pages to save and charge. Small default (20 pages = $0.009) so agent test calls stay cheap; raise it for full sites.

## `maxCrawlDepth` (type: `integer`):

Link depth from the start URLs (0 = only the start URLs).

## `maxTokens` (type: `integer`):

Cut each page to about this many tokens (chars/4 estimate), e.g. `1500` for embedding-sized pages. Overrides maxContentChars. Use when: pages go into embeddings or a context window.

## `knownHashes` (type: `object`):

Map of URL to the `stats.contentHash` from your last crawl. Pages whose content is unchanged are skipped and NOT charged; their links are still followed. Re-crawl a docs site on a schedule and pay only for pages that changed or are new. Use when: you re-crawl a site and only want to pay for changed pages.

## `matchHashesAnyUrl` (type: `boolean`):

Also skip a page if its hash matches any known hash, even under a different URL (moved pages).

## `includeUrlGlobs` (type: `array`):

Optional. Only crawl URLs matching these globs, e.g. `[{"glob": "https://example.com/docs/**"}]`. Overrides the start-URL prefix rule. Use when: you only want one section of a site.

## `excludeUrlGlobs` (type: `array`):

Skip URLs matching these globs, e.g. `[{"glob": "**/changelog/**"}]`.

## `useLlmsTxt` (type: `boolean`):

If the site publishes /llms.txt, enqueue the pages it links (fast, curated coverage of docs sites). Use when: the site has /llms.txt (fast, curated docs coverage).

## `useSitemaps` (type: `boolean`):

Also enqueue in-scope URLs from /sitemap.xml.

## `saveMarkdown` (type: `boolean`):

Return `markdown` per page (recommended for LLMs). `text` is always returned.

## `saveHtml` (type: `boolean`):

Also return cleaned body HTML in `html`.

## `maxContentChars` (type: `integer`):

Cut each page to this many characters to keep payloads small. 0 = no limit.

## `htmlTransformer` (type: `string`):

`readableText` keeps main content only; `none` converts the whole body after removing `removeElementsCssSelector`.

## `removeElementsCssSelector` (type: `string`):

CSS selector for elements removed before conversion.

## `keepUrlFragments` (type: `boolean`):

Treat `#fragment` URLs as separate pages.

## `maxConcurrency` (type: `integer`):

Parallel requests.

## `requestTimeoutSecs` (type: `integer`):

Per-page timeout.

## `maxResults` (type: `integer`):

Optional extra cap on saved pages (same name as apify/website-content-crawler).

## `crawlerType` (type: `string`):

Accepted for apify/website-content-crawler compatibility. This Actor always uses fast raw HTTP (no JavaScript rendering).

## `proxyConfiguration` (type: `object`):

Optional. Off by default. Use when: a site blocks direct requests. Leave off otherwise.

## `debugMode` (type: `boolean`):

Keep Open Graph and JSON-LD in `metadata`.

## Actor input object example

```json
{
  "mode": "crawl",
  "query": "https://docs.apify.com/platform/actors",
  "urls": [
    "https://example.com"
  ],
  "outputFormats": [
    "markdown"
  ],
  "includeLinks": false,
  "maxUrls": 100,
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/"
    }
  ],
  "maxCrawlPages": 20,
  "maxCrawlDepth": 2,
  "matchHashesAnyUrl": false,
  "useLlmsTxt": true,
  "useSitemaps": false,
  "saveMarkdown": true,
  "saveHtml": false,
  "maxContentChars": 30000,
  "htmlTransformer": "readableText",
  "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, iframe, form, img[src^='data:'], [role=\"alert\"], [role=\"banner\"], [role=\"dialog\"], [role=\"alertdialog\"], [aria-modal=\"true\"], .cookie, #cookie, [class*=\"cookie-banner\"], [id*=\"cookie-banner\"]",
  "keepUrlFragments": false,
  "maxConcurrency": 10,
  "requestTimeoutSecs": 30,
  "crawlerType": "cheerio",
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "debugMode": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.python.org/3/tutorial/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("johto.data/site-markdown-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://docs.python.org/3/tutorial/" }] }

# Run the Actor and wait for it to finish
run = client.actor("johto.data/site-markdown-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/"
    }
  ]
}' |
apify call johto.data/site-markdown-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,johto.data/site-markdown-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/XdkX6e4Umk14bekXS/builds/2R6Um2g0ubnYez4XP/openapi.json