# Documentation Site Scraper (`bindler/documentation-site-scraper`) Actor

Scrape any documentation site to clean markdown. Works on Docusaurus, Mintlify, GitBook, MkDocs, ReadTheDocs and more. Preserves code blocks for RAG and LLM training.

- **URL**: https://apify.com/bindler/documentation-site-scraper.md
- **Developed by:** [Neil Sangwaiya](https://apify.com/bindler) (community)
- **Categories:** AI, Developer tools, Agents
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Documentation Site Scraper

Turn **any documentation site** into clean markdown, with code blocks intact. Built for RAG pipelines, AI coding assistants, LLM fine-tuning and docs migration.

Works on Docusaurus, Mintlify, GitBook, MkDocs, ReadTheDocs, Nextra, VuePress, Sphinx and plain HTML docs. No configuration, no framework selection. Paste the URL.

**It also works on help centres**, including Freshdesk, Intercom, HelpScout and Document360, which have no public API. Tested against Freshdesk (1,432 pages discovered) and Intercom (5,606 pages). If your target runs on Zendesk, use the dedicated [Zendesk Help Center Scraper](https://apify.com/bindler/zendesk-help-center-scraper) instead, since Zendesk's public API returns cleaner structured fields than HTML extraction can.

### What you get

| Field | Description |
|---|---|
| `url` | Page URL |
| `title` | The page's own heading, not the site name |
| `description` | Meta description |
| `markdown` | **Full page content as clean markdown** |
| `wordCount` | Words on the page |
| `codeBlocks` | Number of fenced code blocks |
| `headings` | Every heading, in order, for chunking |
| `site` | Source site |
| `scrapedAt` | ISO timestamp |

### Why this one is different

**Code blocks survive.** Syntax highlighters wrap every line in its own element, and a naive text extraction glues them into one unreadable line. Most docs scrapers return `npm installcd my-appnpm run dev`. This one reconstructs the lines and returns a proper fenced block with the language tag. For documentation, mangled code is worse than no code.

**Titles are the page's own heading.** Falling back to the `<title>` tag gives every page the same site-wide name, which makes a dataset useless for retrieval. This reads the real `h1` first, before any markup transformation destroys it.

**Discovery actually works.** Sitemaps come in three shapes: a plain sitemap, an index pointing at more sitemaps, or missing entirely with the real one declared in `robots.txt`. Docs on a subpath often have their own separate sitemap. This handles all of it, and falls back to a scoped link crawl when there is no sitemap at all.

**It scopes to what you asked for.** Give it `example.com/docs` and you get the docs, not the blog, the pricing page and every translation. Root sitemaps list the whole site; this narrows to your path automatically.

**Headings are extracted separately** so you can chunk by section rather than by arbitrary character count, which is the single biggest quality lever in a RAG pipeline.

### Example input

```json
{
  "startUrl": "https://docs.apify.com",
  "maxPages": 500,
  "excludePattern": "/blog/|/changelog/"
}
```

### Filters

- **Only URLs matching** — a regular expression, e.g. `/api/` for just the API reference
- **Skip URLs matching** — e.g. `/blog/|/es/|/fr/` to drop blogs and translations
- **Stay on the same host** — on by default, and treats `www.` as the same site
- **Max pages** — caps the run so cost is predictable

### Notes

- Strips navigation, sidebars, footers, breadcrumbs, cookie prompts and skip links before extraction.
- Retries with backoff and respects rate limits.
- Uses only public pages. No login, no private docs.

# Actor input Schema

## `startUrl` (type: `string`):

Any docs site or help centre, e.g. https://docs.apify.com or https://support.example.com/support/solutions. Works with Docusaurus, Mintlify, GitBook, MkDocs, ReadTheDocs, Nextra, Freshdesk, Intercom and plain HTML.

## `maxPages` (type: `integer`):

Stop after this many pages. Controls your cost.

## `includePattern` (type: `string`):

Optional regular expression. Example: /docs/ to restrict to the docs section.

## `excludePattern` (type: `string`):

Optional regular expression. Example: /blog/|/changelog/

## `sameHostOnly` (type: `boolean`):

Ignore links pointing to other domains.

## Actor input object example

```json
{
  "startUrl": "https://docs.apify.com",
  "maxPages": 200,
  "sameHostOnly": true
}
```

# Actor output Schema

## `pages` (type: `string`):

One record per page with markdown content, headings and code block count.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrl": "https://docs.apify.com"
};

// Run the Actor and wait for it to finish
const run = await client.actor("bindler/documentation-site-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrl": "https://docs.apify.com" }

# Run the Actor and wait for it to finish
run = client.actor("bindler/documentation-site-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrl": "https://docs.apify.com"
}' |
apify call bindler/documentation-site-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,bindler/documentation-site-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/i8LOiu9JJxFXMQH1e/builds/sDCJvRCJ2bGMMevE9/openapi.json