# Markdown Crawler API — Crawl a Whole Website to Markdown (`highbrow_fame/markdown-site-crawler`) Actor

Crawls a site (links or sitemap, depth and page limits, robots.txt respected) into one Markdown document per page with title, URL, headings and links, for RAG. Unofficial Website to Markdown API.

- **URL**: https://apify.com/highbrow_fame/markdown-site-crawler.md
- **Developed by:** [yestrue](https://apify.com/highbrow_fame) (community)
- **Categories:** AI, Agents, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.00 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Markdown Crawler API — Crawl a Whole Website to Markdown

Point it at a website and get one clean Markdown document per page — title, address, depth, headings, text and word count — ready for a vector database, a RAG pipeline or an LLM context window. It follows same-site links breadth first (or reads the sitemap), stays within your page and depth limits, respects `robots.txt` and its crawl delay, and skips images, PDFs and scripts. Navigation, headers, footers and sidebars are dropped with the Readability algorithm, so you store content, not chrome.

**Try it in 1 minute:** press **Start** with the prefilled input (the Apify platform docs, 50 pages, depth 3). Within a minute you get up to 50 records, each with `url`, `depth`, `title`, `markdown`, `text`, `wordCount` and `headings`. Price: $1.00 per 1,000 pages; pages that fail or are blocked by robots.txt are free.

**Use it to:**

- 🧠 **Build a RAG knowledge base** — documentation sites, help centres and blogs turned into Markdown chunks with headings for splitting.
- 🤖 **Give an agent a whole site** — crawl once, hand the Markdown to the model, answer questions about the product.
- 🔍 **Audit content** — every page's title, description, language, word count and link count in one table.
- 🔁 **Keep a mirror fresh** — schedule a daily crawl and re-embed only the pages whose text changed.
- 🗺️ **Map a site** — `depth`, `parentUrl` and `linksCount` show how pages connect.

### How it works

1. Each start URL is fetched over plain HTTP with browser-like headers (30-second timeout, three attempts; a network error is retried through the Apify proxy). `robots.txt` is read once per site: disallowed pages are skipped and the `Crawl-delay` is honoured (up to 10 s).
2. Every `<a href>` on the page that stays on the same website (or matches your **Include only** patterns and none of your **Exclude** patterns) is queued, breadth first, until **Max pages** addresses have been taken or **Max depth** is reached. Hashes and (by default) query strings are ignored, so a page is fetched once. With **Seed from sitemap** the site's sitemap pages are queued too.
3. Each page's main content is isolated with Mozilla Readability and converted to Markdown (headings, lists, links, images and tables kept; addresses made absolute). When no main content is detected, the whole body minus navigation and footer is used — a page is never silently empty.
4. E-mail addresses and phone numbers in the text are removed. Every page is one record with a `status`.

No browser is started, which keeps the crawl fast and cheap. Sites that render only in JavaScript come back with thin pages (`readable: false`, few words); a browser-based version is not included in this release.

### Input

| Field | Meaning | Default |
|---|---|---|
| `startUrls` | Where to begin, one per line | *(required)* |
| `maxPages` | Pages fetched per start URL | `50` |
| `maxDepth` | Links away from the start URL | `3` |
| `includeGlobs` / `excludeGlobs` | Address patterns to follow / never follow (`**` anything, `*` anything but `/`) | none |
| `useSitemap` | Also queue the site's sitemap pages | `false` |
| `respectRobotsTxt` | Skip disallowed pages, honour Crawl-delay | `true` |
| `sameDomainOnly` | Stay on the start URL's host (www. ignored) | `true` |
| `stripQuery` | Treat `/page?x=1` and `/page` as one page | `true` |
| `ignoreExtensions` | File types never fetched | images, PDF, archives, media, CSS, JS, … |
| `mainContentOnly` | Readability main content vs. whole page | `true` |
| `includeLinks` | Deliver each page's links (up to 500) | `false` |
| `maxTextLength` | Cut text and Markdown after N characters | `0` (no limit) |
| `maxConcurrency` | Pages fetched in parallel | `5` |
| `proxyConfiguration` | Proxy for retries after a network error | Apify proxy |

### Output

| Field | What it is |
|---|---|
| `url`, `finalUrl`, `canonicalUrl` | the page as queued, after redirects, and the canonical address it declares |
| `depth`, `parentUrl` | how far from the start URL, and on which page the link was found (`sitemap` for sitemap pages) |
| `title`, `description`, `language` | page metadata as the page declares it |
| `markdown`, `text` | the content as Markdown (title as first heading) and as plain text |
| `wordCount`, `headings`, `readable` | size, heading outline, and whether a main content block was found |
| `linksCount`, `links` | unique links on the page, and the links themselves with `includeLinks` |
| `input`, `status`, `error`, `httpStatus`, `scrapedAt` | the start URL, `OK` or a reason code, and when |

Statuses: `OK` (charged) — `NOT_FOUND`, `NOT_HTML`, `TOO_LARGE`, `ROBOTS_DISALLOWED`, `INVALID_INPUT`, `FAILED` (free, with `error` saying why). Pages that robots.txt disallows are skipped without a record, except a start page, which gets a `ROBOTS_DISALLOWED` record so you know why the run is empty.

#### Example

```json
{
  "input": "https://docs.example.com/guide",
  "url": "https://docs.example.com/guide/getting-started",
  "finalUrl": "https://docs.example.com/guide/getting-started",
  "depth": 1,
  "parentUrl": "https://docs.example.com/guide",
  "title": "Getting started",
  "description": "Install the CLI and run your first job.",
  "language": "en",
  "canonicalUrl": "https://docs.example.com/guide/getting-started",
  "markdown": "# Getting started\n\nInstall the CLI with your package manager …\n\n## Your first job\n\n1. Create a project\n2. Run `example start`\n\n| Flag | Meaning |\n| --- | --- |\n| `--watch` | rerun on change |",
  "text": "Install the CLI with your package manager …",
  "wordCount": 412,
  "readable": true,
  "headings": [{ "level": 1, "text": "Getting started" }, { "level": 2, "text": "Your first job" }],
  "linksCount": 38,
  "httpStatus": 200,
  "status": "OK",
  "scrapedAt": "2026-10-19T12:00:00.000Z"
}
```

### Pricing

You pay **per page delivered** ($1.00 per 1,000). Pages that fail, are not HTML or are blocked by robots.txt are free. **Max pages** bounds each crawl, and the run's **Maximum charge** caps the cost: the run stops cleanly when it is reached and tells you so.

### FAQ

**Is this an official API?** No, it's an unofficial API built on public pages.

**How do I crawl only one section of a site?** Put a pattern in **Include only**, e.g. `https://docs.example.com/guide/**`. Only addresses matching it are followed; the start URL is always fetched.

**Why are there fewer pages than Max pages?** The site has fewer reachable pages within **Max depth**, links were excluded by your patterns or by robots.txt, or they point to files (PDF, images) that are skipped. `linksCount` on each record shows what was found.

**Why does a page have only a few words?** It is built in JavaScript and ships an empty shell; a plain fetch sees nothing to convert. Such pages come back with `readable: false`. A headless-browser mode is planned.

**Does it run JavaScript or log in?** No. It reads what a logged-out visitor without JavaScript sees and never bypasses captchas, logins or bot walls.

**Like it?** A short review on the Store page helps other people find this Actor. Something missing or broken? Tell us on the Issues tab — we read every one.

### Related Actors

- [Sitemap Extractor & Scraper](https://apify.com/highbrow_fame/sitemap-url-extractor) — every URL, date and image from a sitemap
- [Website Tech Stack Detector](https://apify.com/highbrow_fame/website-tech-stack) — CMS, frameworks and analytics of any site
- [URL to Markdown API](https://apify.com/highbrow_fame/url-to-markdown-api) — clean Markdown from any URL for LLMs
- [Article Extractor API](https://apify.com/highbrow_fame/article-extractor-api) — title, author, date and full text of any article
- [PDF to Markdown API](https://apify.com/highbrow_fame/pdf-to-markdown-api) — PDFs to Markdown with headings and tables

# Actor input Schema

## `startUrls` (type: `array`):

Where the crawl begins, one address per line. Links on each page that stay on the same website are followed, breadth first, until the page or depth limit is reached.

## `maxPages` (type: `integer`):

Stop after this many pages per start URL (fetched pages; failed pages count too). The run's "Maximum charge" caps the cost as well.

## `maxDepth` (type: `integer`):

How many links away from the start URL to go. 0: only the start pages; 1: the start pages and the pages they link to; and so on.

## `includeGlobs` (type: `array`):

Follow only addresses matching one of these patterns, e.g. https://docs.example.com/guide/\*\* or /blog/\*\*. \*\* matches anything, \* anything but a slash. Empty: every page on the same website.

## `excludeGlobs` (type: `array`):

Never follow addresses matching one of these patterns, e.g. **/tag/** or https://docs.example.com/archive/\*\*.

## `useSitemap` (type: `boolean`):

Also read the site's sitemap (from robots.txt or /sitemap.xml, one index level deep) and queue its pages at depth 1 — the fastest way to cover a whole site.

## `respectRobotsTxt` (type: `boolean`):

Skip pages the site's robots.txt disallows and honour its Crawl-delay (up to 10 s). Turn off only for sites you have permission to crawl.

## `sameDomainOnly` (type: `boolean`):

Follow only links on the start URL's host (www. and non-www. count as one). Turn off to follow links anywhere — then use Include only to keep the crawl bounded.

## `stripQuery` (type: `boolean`):

Treat /page?utm=x and /page as the same address and drop the query. Turn off for sites where the query selects the content (?id=123).

## `ignoreExtensions` (type: `array`):

Links ending in one of these extensions are not fetched.

## `mainContentOnly` (type: `boolean`):

Keep the main content of each page (Readability) and drop navigation, headers, footers and sidebars. Turn off to convert the whole page.

## `includeLinks` (type: `boolean`):

Also deliver every link on each page (up to 500) as links\[] with href and anchor text.

## `maxTextLength` (type: `integer`):

Cut text and Markdown after this many characters (textTruncated: true). 0: no limit.

## `maxConcurrency` (type: `integer`):

How many pages to fetch at the same time. A site's Crawl-delay still applies between its pages.

## `proxyConfiguration` (type: `object`):

Used only to try again after a network error: the first try goes out directly.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com/platform"
  ],
  "maxPages": 50,
  "maxDepth": 3,
  "includeGlobs": [],
  "excludeGlobs": [],
  "useSitemap": false,
  "respectRobotsTxt": true,
  "sameDomainOnly": true,
  "stripQuery": true,
  "ignoreExtensions": [
    "pdf",
    "jpg",
    "jpeg",
    "png",
    "gif",
    "webp",
    "svg",
    "ico",
    "bmp",
    "tif",
    "tiff",
    "zip",
    "gz",
    "tgz",
    "tar",
    "rar",
    "7z",
    "mp3",
    "mp4",
    "m4a",
    "m4v",
    "mov",
    "avi",
    "webm",
    "ogg",
    "wav",
    "css",
    "js",
    "mjs",
    "json",
    "xml",
    "rss",
    "atom",
    "woff",
    "woff2",
    "ttf",
    "eot",
    "otf",
    "doc",
    "docx",
    "xls",
    "xlsx",
    "ppt",
    "pptx",
    "exe",
    "dmg",
    "apk",
    "iso",
    "bin"
  ],
  "mainContentOnly": true,
  "includeLinks": false,
  "maxTextLength": 0,
  "maxConcurrency": 5,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

Every result as a JSON record, with a status for each input.

## `resultsCsv` (type: `string`):

The same records as CSV, for a spreadsheet.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com/platform"
    ],
    "includeGlobs": [],
    "excludeGlobs": [],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("highbrow_fame/markdown-site-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.apify.com/platform"],
    "includeGlobs": [],
    "excludeGlobs": [],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("highbrow_fame/markdown-site-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com/platform"
  ],
  "includeGlobs": [],
  "excludeGlobs": [],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call highbrow_fame/markdown-site-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,highbrow_fame/markdown-site-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/3uRUVEda4CVxdHmSy/builds/DuFsuNYcyz6iRckpt/openapi.json