# Website to Markdown Crawler for LLMs and RAG (`fecklesspleb/website-to-markdown-crawler`) Actor

Crawl a whole website, docs section or blog and save every page as clean Markdown or text for LLMs, RAG, vector databases and AI chatbots. Depth, page and path limits, include and exclude globs. Stays on the site, respects robots.txt, no browser. $0.80 per 1,000 pages.

- **URL**: https://apify.com/fecklesspleb/website-to-markdown-crawler.md
- **Developed by:** [Scott Williams](https://apify.com/fecklesspleb) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown Crawler for LLMs and RAG

Crawl a website, a documentation section or a blog and get every page as clean Markdown or plain text, ready to load into a vector database, a custom GPT, an AI chatbot or any RAG pipeline. Give it one or more start URLs; it follows links on the same site, extracts the main content of each page (menus, footers, cookie banners and ads removed), and saves one dataset row per page with the URL, title, description, language, canonical URL, link depth and Markdown.

**At a glance:** $0.80 per 1,000 pages, plus $0.00005 per run start. Pay per result with no subscription, set a maximum cost per run, export to JSON, CSV or Excel, or call it from the Apify API, Make, Zapier or n8n.

Plain HTTP with no browser, so it is fast and cheap. robots.txt is respected (RFC 9309), including crawl delays, on every request and redirect.

### Use cases

- Build a knowledge base for a support chatbot from your help center or docs.
- Load a competitor's public docs or a product's changelog into an LLM for analysis (where their terms allow it).
- Migrate a website's content to Markdown for a static site generator.
- Create a text snapshot of a site for search, auditing or change tracking.

### How the crawl is scoped

- Only the same host as the start URL is followed (`www.` is ignored).
- With **Stay within the start URL's folder** on (default), `https://example.com/docs/` crawls only `/docs/...`.
- Optional include and exclude globs, for example `*/blog/*` or `*?page=*`.
- Images, PDFs, archives, media, CSS and JS links are skipped.
- Tracking parameters (`utm_*`, `gclid`, `fbclid`) and `#fragments` are removed before de-duplication.

### Input

```json
{
  "startUrls": [{"url": "https://docs.python.org/3/tutorial/"}],
  "maxPages": 10,
  "maxDepth": 5,
  "stayWithinPath": true,
  "contentMode": "main",
  "outputFormat": "markdown"
}
```

### Output example

Real output from a 5-page test crawl of the URL above (Markdown shortened):

```json
{
  "url": "https://docs.python.org/3/tutorial/appetite.html",
  "depth": 1,
  "referrerUrl": "https://docs.python.org/3/tutorial/",
  "finalUrl": "https://docs.python.org/3/tutorial/appetite.html",
  "statusCode": 200,
  "title": "1. Whetting Your Appetite — Python 3.15.2 documentation",
  "description": "If you do much work on computers, eventually you find that there’s some task you’d like to automate. ...",
  "language": "en",
  "canonicalUrl": "https://docs.python.org/3/tutorial/appetite.html",
  "markdown": "# 1. Whetting Your Appetite\n\nIf you do much work on computers, eventually you find ...",
  "wordCount": 56,
  "truncated": true,
  "error": null,
  "crawledAt": "2026-10-28T03:50:00+00:00"
}
```

A `RUN_SUMMARY` record in the key-value store lists pages saved, failures, URLs blocked by robots.txt and URLs discovered.

### Pricing

Pay per saved page: **$0.80 per 1,000 pages** ($0.0008 per page), plus the standard Apify run start event of $0.00005 per run. Failed, skipped and robots-blocked pages are free. **Max pages** caps every run, and **Maximum cost per run** stops it cleanly.

### Use with AI agents / MCP

AI agents can run this Actor as a tool through [Apify's MCP server](https://docs.apify.com/platform/integrations/mcp). Add this server URL to your MCP client (Claude, Cursor, VS Code, OpenAI Agents SDK, LangChain and others) and sign in with OAuth, or send your Apify token as `Authorization: Bearer <token>`:

```text
https://mcp.apify.com?tools=fecklesspleb/website-to-markdown-crawler
```

The tool is named `fecklesspleb--website-to-markdown-crawler`. Agents can also find it with `search-actors` and run it with `call-actor`. The run returns a dataset ID; read the rows with `get-dataset-items`, and pass `fields` (for example `url,depth,title,markdown`) to keep the response small.

Minimal tool input:

```json
{"startUrls": [{"url": "https://docs.python.org/3/tutorial/"}], "maxPages": 3, "maxCharsPerPage": 300}
```

This input saves 3 rows: the start page (`depth` 0, `referrerUrl` null) and two pages it links to (`depth` 1). Each row has the fields shown in the Output example above.

Tips for agents:

- Cost is $0.0008 per saved page. `maxPages` (default 10) caps every run, so set it to what you need, and set `maxTotalChargeUsd` in the run options (`callOptions` in `call-actor`) as a second cap.
- Start from the folder you want: with the default `stayWithinPath: true`, `https://example.com/docs/` crawls only `/docs/...`. Use `excludeUrlGlobs` to skip tag, archive or paging URLs.
- If no page is saved, the dataset holds one free row with `recordType: "status"` and a `message` saying why (failures, robots.txt blocks).
- The `RUN_SUMMARY` record in the run's key-value store shows `pagesSaved`, `failed`, `skippedRobots`, `urlsDiscovered` and `stoppedAtMaxPages`.

### Limits

- No JavaScript rendering, so sites that build their content in the browser may return little text. Server-rendered sites (most docs, blogs, news, marketing sites) work well.
- No login support.
- On the Apify platform, pages whose address is private, loopback or link-local (also after a redirect) are not crawled and are free. They count as failed pages (`error: "not fetched: private, loopback and link-local addresses are not allowed"` when `includeFailedPages` is on).

### Is this legal?

The crawler identifies itself with its own user agent, obeys robots.txt and crawl delays, and spaces requests to each host. It only reads public pages. You are responsible for using the content in line with each site's terms and copyright.

# Changelog

This Actor's version history is a separate document: https://apify.com/fecklesspleb/website-to-markdown-crawler/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

Where the crawl begins, as objects: \[{"url": "https://docs.python.org/3/tutorial/"}]. Links are followed on the same host only (www. is ignored). A URL without http(s):// gets https:// added. Start URLs are always fetched; scope rules and globs apply to the links found on them.

## `maxPages` (type: `integer`):

Stop after saving this many pages. Default 10. This is the cost cap: at most maxPages x $0.0008 in page events per run. Failed, skipped and robots-blocked pages are free and are not counted.

## `maxDepth` (type: `integer`):

How many clicks away from a start URL to follow. Default 5. 0 saves only the start URLs.

## `stayWithinPath` (type: `boolean`):

If true (default), only URLs under the start URL's folder are crawled: with https://example.com/docs/ only /docs/... pages. Set false to crawl the whole host.

## `includeUrlGlobs` (type: `array`):

Optional glob patterns matched against the full URL (tracking parameters and #fragments removed), e.g. \["https://example.com/blog/\*"]. When set, only matching links are queued, so pages reachable only through non-matching pages are not found. Leave empty to include everything in scope.

## `excludeUrlGlobs` (type: `array`):

Glob patterns matched against the full URL; matching links are not queued, e.g. \["*/tag/*", "*?page=*"].

## `contentMode` (type: `string`):

"main" (default): main content only, boilerplate removed with trafilatura, with a whole-page fallback. "page": whole page without nav, header and footer. "full": whole page including navigation.

## `outputFormat` (type: `string`):

"markdown" (default) puts the content in the markdown field, keeping headings, lists, tables and links. "text" puts plain text in the text field instead.

## `maxCharsPerPage` (type: `integer`):

Cut the markdown or text of each page to this many characters, e.g. 4000; truncated is then true. Default 0 = no limit.

## `includeFailedPages` (type: `boolean`):

If true, pages that returned HTTP 400 or higher, or failed to load, get a free row with error set, useful for spotting broken links. Pages blocked by robots.txt, and hosts that could not be reached at all, get no row. Default false.

## `maxConcurrency` (type: `integer`):

Parallel workers, 1 to 20. Default 5. Requests to one host are spaced at least 0.5 s apart, or by the robots.txt crawl delay.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/"
    }
  ],
  "maxPages": 10,
  "maxDepth": 5,
  "stayWithinPath": true,
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "contentMode": "main",
  "outputFormat": "markdown",
  "maxCharsPerPage": 0,
  "includeFailedPages": false,
  "maxConcurrency": 5
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.python.org/3/tutorial/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("fecklesspleb/website-to-markdown-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://docs.python.org/3/tutorial/" }] }

# Run the Actor and wait for it to finish
run = client.actor("fecklesspleb/website-to-markdown-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/"
    }
  ]
}' |
apify call fecklesspleb/website-to-markdown-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,fecklesspleb/website-to-markdown-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/pVMxDbeMGHZzqbKb0/builds/EPjVqdFjK76y53HMw/openapi.json