# Website to Markdown for LLM & RAG: Fast Crawler, No Browser (`blond_terrarium/website-to-markdown-llm`) Actor

Crawl any public website and get clean, LLM-ready Markdown: main content only, tables and code kept, metadata, token estimates and optional RAG chunks. Fast HTTP crawling, robots.txt respected. Pay only per page converted; failed or empty pages are free.

- **URL**: https://apify.com/blond_terrarium/website-to-markdown-llm.md
- **Developed by:** [Mogul Tools](https://apify.com/blond_terrarium) (community)
- **Categories:** AI, Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 page converted to markdowns

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown for LLM & RAG: Fast Crawler, No Browser

Turn any public website into **clean, LLM-ready Markdown** in seconds. Give it one page or a whole docs site; get back the main content (no menus, cookie banners or footers) with headings, lists, **tables** and **code blocks with language tags** preserved, plus title, description, author, date, language, a token estimate, a content hash and optional **ready-made RAG chunks**.

It uses fast HTTP requests instead of a headless browser, so it is quick and cheap: about **500 pages in under a minute** in our tests.

**You only pay for pages that were actually converted.** 404s, timeouts, anti-bot pages, empty pages, PDFs/images and robots.txt-blocked URLs are free.

### What you can use it for

- Feeding documentation, help centers, blogs and knowledge bases into a **vector database / RAG pipeline** (LangChain, LlamaIndex, Pinecone, OpenAI Assistants, custom GPTs)
- Giving **AI agents** a clean Markdown view of any web page (works via the Apify MCP server)
- Building training or fine-tuning datasets from public sites
- Content audits and **change detection** (compare `contentHash` between runs)

### Features

- **Main-content extraction** (Mozilla Readability, with a sanity check against `<article>`/`<main>`), or whole-page mode
- GitHub-flavored Markdown: tables, strikethrough, fenced code with language
- Absolute links and image URLs; options to drop images or links to save tokens
- **Crawling**: depth limit, page limit, scope (same path / hostname / domain), include/exclude regex, optional **sitemap discovery** (robots.txt, sitemap indexes, .gz)
- **RAG chunking** on heading/paragraph boundaries with overlap (no extra charge)
- Polite by default: respects robots.txt and Crawl-delay, per-host delay, retries with backoff and Retry-After
- Fair crawling across multiple start URLs (one big site can't starve the others)
- Handles redirects (incl. broken https→http redirects), charsets, `<base href>`, canonical URLs, text/markdown responses
- Refuses private/internal addresses
- Stops cleanly at your **max total charge** without storing unpaid results

### Input example

```json
{
  "startUrls": ["https://docs.python.org/3/tutorial/"],
  "maxCrawlDepth": 2,
  "maxPages": 200,
  "crawlScope": "samePath",
  "chunkSize": 2000
}
```

### Output example (one row per page)

```json
{
  "url": "https://docs.python.org/3/tutorial/appetite.html",
  "loadedUrl": "https://docs.python.org/3/tutorial/appetite.html",
  "statusCode": 200,
  "depth": 1,
  "title": "1. Whetting Your Appetite",
  "language": "en",
  "extractionMode": "article",
  "markdown": "# 1. Whetting Your Appetite\n\nIf you do much work on computers, eventually you find that there's some task you'd like to automate...",
  "wordCount": 730,
  "charCount": 4581,
  "tokensEstimate": 1146,
  "contentHash": "4f1c0e...",
  "chunks": [{ "index": 0, "text": "# 1. Whetting Your Appetite\n\n..." }],
  "crawledAt": "2026-10-22T02:30:11.000Z"
}
```

Failed and skipped URLs (with the reason) are listed in the `RUN_SUMMARY` record of the key-value store. They are not charged.

### Pricing (pay per event)

You pay per **page converted** (a page that returned HTTP 2xx with at least `minContentChars` of text):

| Apify plan | Price per 1,000 pages |
|---|---|
| Free | $2.00 |
| Bronze (Starter) | $1.70 |
| Silver (Scale) | $1.40 |
| Gold (Business) | $1.00 |

Platform usage (compute, storage) is included; you don't pay it separately. Each run also has Apify's standard start fee of $0.00005 per GB of memory. Optional Apify datacenter proxy (or your own proxy URLs) is included in the price; residential proxy isn't supported. Set **Max total charge** on the run to cap your spend; the run stops cleanly when it's reached.

### Limitations

- No JavaScript rendering. Sites that build their content in the browser (some SPAs, client-side blogs) may return little text; such pages are marked empty and **not charged**.
- Sites with strong bot protection (Cloudflare challenge, Akamai) may block requests; those pages are free. Enabling Apify datacenter proxy sometimes helps.
- PDFs and other files are skipped (free).

### Responsible use

Only crawl public pages you are allowed to process. robots.txt is respected by default. Don't use the output to republish copyrighted content without permission.

# Actor input Schema

## `startUrls` (type: `array`):

Pages or sites to convert. 'example.com' is fine; https:// is added automatically.

## `maxCrawlDepth` (type: `integer`):

0 = convert only the start URLs. 1 = also pages linked from them, and so on.

## `maxPages` (type: `integer`):

Stop after this many converted pages in total (0 = no limit). Your max total charge also stops the run cleanly.

## `crawlScope` (type: `string`):

Which discovered links to follow.

## `useSitemaps` (type: `boolean`):

Reads robots.txt / sitemap.xml (incl. indexes and .gz) to find pages that aren't linked.

## `includeUrlPatterns` (type: `array`):

Only follow/convert URLs matching at least one pattern, e.g. '/blog/'.

## `excludeUrlPatterns` (type: `array`):

Skip URLs matching any pattern, e.g. '/tag/' or '?page='.

## `contentMode` (type: `string`):

How to pick the content to convert.

## `removeSelectors` (type: `string`):

Comma-separated, e.g. '.related-posts, #comments'.

## `keepImages` (type: `boolean`):

Image references as ![alt](absolute-url).

## `keepLinks` (type: `boolean`):

Turn off to get plain text without link URLs (fewer tokens).

## `chunkSize` (type: `integer`):

If > 0, each page also gets a 'chunks' array split on headings/paragraphs. Free: chunks don't add charges.

## `chunkOverlap` (type: `integer`):

Characters repeated between consecutive chunks.

## `includeText` (type: `boolean`):

Adds a 'text' field.

## `includeHtml` (type: `boolean`):

Adds an 'html' field with the extracted content HTML.

## `includeLinksList` (type: `boolean`):

Adds a 'links' array (absolute URLs).

## `minContentChars` (type: `integer`):

Pages with less text are treated as empty and are not charged.

## `respectRobotsTxt` (type: `boolean`):

Skip disallowed URLs and honor Crawl-delay (max 10 s).

## `maxConcurrency` (type: `integer`):

Across all hosts. Requests to one host are always spaced by the delay below.

## `requestDelayMs` (type: `integer`):

Politeness per host.

## `requestTimeoutSecs` (type: `integer`):

Per request.

## `maxRetries` (type: `integer`):

For timeouts, resets, 429 and 5xx (with backoff and Retry-After).

## `userAgent` (type: `string`):

Leave empty for the default browser-compatible UA.

## `proxyConfiguration` (type: `object`):

Optional. Apify datacenter proxy or your own proxy URLs. Residential proxy isn't supported (it would raise the price for everyone).

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.python.org/3/tutorial/"
  ],
  "maxCrawlDepth": 1,
  "maxPages": 100,
  "crawlScope": "samePath",
  "useSitemaps": false,
  "includeUrlPatterns": [],
  "excludeUrlPatterns": [],
  "contentMode": "article",
  "keepImages": true,
  "keepLinks": true,
  "chunkSize": 0,
  "chunkOverlap": 200,
  "includeText": false,
  "includeHtml": false,
  "includeLinksList": false,
  "minContentChars": 100,
  "respectRobotsTxt": true,
  "maxConcurrency": 10,
  "requestDelayMs": 200,
  "requestTimeoutSecs": 30,
  "maxRetries": 2,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `overview` (type: `string`):

Converted pages in the 'Pages' overview view.

## `results` (type: `string`):

Full dataset rows including Markdown and chunks.

## `summary` (type: `string`):

Totals and failed URLs (not charged).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.python.org/3/tutorial/"
    ],
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("blond_terrarium/website-to-markdown-llm").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.python.org/3/tutorial/"],
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("blond_terrarium/website-to-markdown-llm").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.python.org/3/tutorial/"
  ],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call blond_terrarium/website-to-markdown-llm --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,blond_terrarium/website-to-markdown-llm"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/z7NXtDCMqlgPCUuph/builds/zwvDgy07bMqwjxcjB/openapi.json