# Website to Markdown for AI/LLM — Cheap RAG Crawler (`fabrikit/website-to-markdown-scraper`) Actor

Crawl any website over plain HTTP (no browser) and get clean Markdown for LLMs and RAG: boilerplate removed, headings outline, optional chunks with overlap, sitemap and robots.txt support, JS-only pages flagged. From US$0.50 per 1,000 pages.

- **URL**: https://apify.com/fabrikit/website-to-markdown-scraper.md
- **Developed by:** [Fabrikit](https://apify.com/fabrikit) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 page converteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown for AI/LLM — Cheap RAG Crawler

Turn any website into clean, LLM-ready Markdown. The Actor crawls pages over **plain HTTP (no browser)**, strips navigation, footers, cookie banners and ads, and returns Markdown (plus optional plain text, cleaned HTML and RAG chunks) with metadata. Because it never starts a browser, it is fast and costs **US$0.50 per 1,000 pages**.

### What you get per page

- `markdown` — main content as clean Markdown (headings, lists, tables, fenced code with language, links, optional images)
- `text` / `html` — optional plain text and cleaned HTML
- `title`, `description`, `language`, `canonicalUrl`
- `headings` — outline with level and text
- `wordCount`, `textLength`, `linksCount`, `depth`, `statusCode`, `fetchedAt`
- `chunks[]` — optional RAG chunks (`index`, `text`, `charStart`, `charEnd`, `heading`) with configurable size and overlap
- `needsBrowser` — `true` when the page is an empty JavaScript shell (single-page app). These pages are returned (not failed) and **not charged**, so you know which URLs need a browser-based crawler.

### Features

- Single pages or same-domain crawl with `maxDepth` and `maxItems` (default 20 pages)
- Include / exclude URL patterns (`https://example.com/docs/**`)
- `sitemap.xml` discovery (from robots.txt or `/sitemap.xml`, sitemap indexes and `.gz` supported)
- Respects `robots.txt` by default (switch off if you own the site)
- Readability-style main-content detection; layout tables unwrapped; real tables kept as Markdown tables
- Handles non-UTF-8 pages (charset detection), redirects, canonical-URL de-duplication
- Automatic proxy escalation: datacenter first, residential only when a site blocks you

### Input example

```json
{
  "startUrls": [{ "url": "https://docs.python.org/3/tutorial/index.html" }],
  "crawlMode": "crawl",
  "maxDepth": 1,
  "maxItems": 20,
  "outputFormats": ["markdown"],
  "enableChunking": true,
  "chunkSize": 1000,
  "chunkOverlap": 100
}
```

### Output example

```json
{
  "url": "https://docs.python.org/3/tutorial/index.html",
  "canonicalUrl": null,
  "title": "The Python Tutorial — Python 3.15.0 documentation",
  "description": "Python is an easy to learn, powerful programming language...",
  "language": "en",
  "markdown": "# The Python Tutorial\n\nPython is an easy to learn...",
  "text": null,
  "html": null,
  "textLength": 6283,
  "wordCount": 1067,
  "headings": [{ "level": 1, "text": "The Python Tutorial" }],
  "linksCount": 174,
  "depth": 0,
  "needsBrowser": false,
  "statusCode": 200,
  "fetchedAt": "2026-10-13T10:00:00.000Z",
  "chunks": [{ "index": 0, "text": "# The Python Tutorial ...", "charStart": 0, "charEnd": 996, "heading": "The Python Tutorial" }],
  "chunksCount": 18,
  "truncated": false,
  "error": null
}
```

### Pricing

| Event | Price | When |
|---|---|---|
| `page-result` | US$0.0005 (US$0.50 / 1,000) | per page converted and saved |
| Actor start | US$0.00005 | once per run |

Pages that need a browser (`needsBrowser: true`), failed pages and robots-blocked pages are not charged.

### FAQ

**Does it run JavaScript?** No. It reads the HTML the server sends, which covers docs sites, blogs, news, wikis, shops' content pages and most marketing sites. Pure single-page apps come back flagged `needsBrowser: true`.

**How do I feed it to an LLM or vector DB?** Use `markdown`, or enable chunking and embed each `chunks[].text`; `heading` gives you the section for metadata.

**What about huge pages?** Fields are capped at 1.5 million characters per page, and pages with more than about 3 MB of HTML per GB of run memory are converted up to that size (links further down are still followed); `truncated` tells you when either happened. Responses over 8 MB are skipped.

**Is it allowed to crawl a site?** Check the site's terms and robots.txt. The Actor respects robots.txt by default.

**Use cases:** RAG knowledge bases from documentation, LLM training/eval corpora, site migration to Markdown, content audits, changelog and blog monitoring.

### En español

Convierte sitios web en Markdown limpio para IA/RAG usando solo HTTP (sin navegador): quita menús, pies de página y banners de cookies; entrega título, descripción, idioma, esquema de encabezados, conteo de palabras y, opcionalmente, trozos (chunks) con solapamiento para embeddings. Soporta crawl del mismo dominio, sitemap.xml y robots.txt. Las páginas que solo funcionan con JavaScript se marcan `needsBrowser: true` y no se cobran. Precio: US$0,50 por 1.000 páginas.

# Actor input Schema

## `startUrls` (type: `array`):

Pages or sites to convert. For a single page use the page URL; for a whole site use the homepage or a docs section URL.

## `crawlMode` (type: `string`):

Single pages: convert only the start URLs. Same-domain crawl: also follow links on the same site.

## `maxItems` (type: `integer`):

Maximum number of pages to convert (billing is per page, JS-only pages are free).

## `maxDepth` (type: `integer`):

How many link levels to follow from the start URLs (0 = start URLs only). Only used by the crawl mode.

## `includeGlobs` (type: `array`):

Only crawl URLs matching one of these patterns, e.g. https://example.com/docs/\*\*. \* matches within a path segment, \*\* matches anything. Start URLs are always converted.

## `excludeGlobs` (type: `array`):

Skip URLs matching any of these patterns, e.g. **/tag/** or \**/login*.

## `useSitemap` (type: `boolean`):

Also read the site's sitemap (from robots.txt or /sitemap.xml) to find pages that are not linked from the start page.

## `respectRobotsTxt` (type: `boolean`):

Skip pages disallowed for crawlers in robots.txt.

## `outputFormats` (type: `array`):

Which content fields to fill in: markdown, plain text and/or cleaned HTML.

## `removeBoilerplate` (type: `boolean`):

Keep only the main content of each page. Turn off to convert the whole page body.

## `keepLinks` (type: `boolean`):

Keep hyperlinks as [text](url). Turn off to get plain text.

## `keepImages` (type: `boolean`):

Keep images as ![alt](url).

## `enableChunking` (type: `boolean`):

Add a chunks array to each page, ready for embedding. Chunks follow paragraph and sentence boundaries and carry the section heading.

## `chunkSize` (type: `integer`):

Maximum characters per chunk.

## `chunkOverlap` (type: `integer`):

Characters repeated between consecutive chunks. Must be smaller than the chunk size.

## `concurrency` (type: `integer`):

Pages fetched in parallel.

## `proxyConfiguration` (type: `object`):

Datacenter proxy by default. If a site blocks it, the Actor retries through a residential proxy automatically.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/index.html"
    }
  ],
  "crawlMode": "crawl",
  "maxItems": 20,
  "maxDepth": 1,
  "includeGlobs": [],
  "excludeGlobs": [],
  "useSitemap": false,
  "respectRobotsTxt": true,
  "outputFormats": [
    "markdown"
  ],
  "removeBoilerplate": true,
  "keepLinks": true,
  "keepImages": false,
  "enableChunking": false,
  "chunkSize": 1000,
  "chunkOverlap": 100,
  "concurrency": 8,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `pages` (type: `string`):

One item per page

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.python.org/3/tutorial/index.html"
        }
    ],
    "maxItems": 20,
    "outputFormats": [
        "markdown"
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("fabrikit/website-to-markdown-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.python.org/3/tutorial/index.html" }],
    "maxItems": 20,
    "outputFormats": ["markdown"],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("fabrikit/website-to-markdown-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/index.html"
    }
  ],
  "maxItems": 20,
  "outputFormats": [
    "markdown"
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call fabrikit/website-to-markdown-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,fabrikit/website-to-markdown-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/woSdfL4G2kz74V4Mh/builds/l5fme7yGuPpjc7eX2/openapi.json