# Website to Markdown Converter for LLM Prompts (`bepolabs/web-to-markdown`) Actor

Paste URLs, get Markdown rows without menus, footers or cookie banners. Optional ready-made chunks for vector search and a depth setting to follow internal hyperlinks.

- **URL**: https://apify.com/bepolabs/web-to-markdown.md
- **Developed by:** [BEPO Labs](https://apify.com/bepolabs) (community)
- **Categories:** AI, Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.40 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### 📝 What is Website to Markdown?

Reading public HTML, Website to Markdown returns the readable body as Markdown, ready for any LLM, retrieval setup or agent. Tables, code, quotes and images keep their structure, with absolute addresses. Name, summary, language and timestamp come along for each result.

- **RAG and vector databases**: load documentation, help centers or blogs into Pinecone, Qdrant, pgvector or LangChain as ready-made chunks
- **Prompting and AI agents**: give Claude, ChatGPT or Cursor the readable text of any page instead of raw HTML
- **Fine-tuning and datasets**: collect clean, structured text of whole sites with word counts for filtering
- **Content and SEO analysis**: compare headings, word counts and links across your own and competitors' pages
- **Archiving and migration**: move pages of an old CMS to Markdown for a static site or a wiki

![Website to Markdown: Any web page or docs site to clean Markdown for LLMs, RAG and AI agents, with headings, links and ready-made chunks.](https://api.apify.com/v2/key-value-stores/wdeAWGooHdIhvFgSY/records/web-to-markdown-hero.png)

#### ✨ Why use this Actor?

- ✅ **Only the main content**: picks `<main>`, `role=main` or the longest `<article>`, and removes nav, header, footer, aside, cookie and newsletter boxes, share bars and hidden elements
- ✅ **RAG chunks built in**: switch it on and every page is also cut at sections, then paragraphs, into pieces of up to `chunkSize` characters, each with its heading path and word count
- ✅ **Crawl a whole section**: follow links on the same host to the depth you set, with include and exclude globs such as `https://example.com/docs/**`
- ✅ **Polite crawling**: robots.txt honoured (can be switched off for your own sites), `noindex` pages skipped and `nofollow` links not followed in crawls, retries on 429 and 5xx
- ✅ **Light on resources**: plain HTTP requests and a built-in HTML parser, no browser to start and no proxy costs
- ✅ **Pay as you go**: 1 USD per 1,000 pages; failed and skipped ones cost nothing

### 📦 What data can you get with Website to Markdown?

| | | |
|---|---|---|
| 📝 Markdown of the main content | 🏷️ Page title | 💬 Meta description |
| 🌐 Language | 🧭 Headings with level | 🔗 Links in the content, with link text |
| 🔢 Word count | 🧩 RAG chunks with heading path | ↪️ Final URL after redirects |
| ✅ HTTP status code | 🕸️ Crawl depth | 🕒 Crawl time |

### 🚀 How to use Website to Markdown

Website to Markdown works in a few clicks, no coding needed:

1. [Create a free Apify account](https://console.apify.com/sign-up) (the free plan includes 5 USD of monthly credit).
2. Open [Website to Markdown](https://apify.com/bepolabs/web-to-markdown) and click **Try for free**.
3. Paste the URLs of the pages you want. To convert a whole section, set a crawl depth of 1 or more and an include glob. Switch on RAG chunks if you feed a vector database.
4. Click **Start** and wait for the run to finish (most runs take seconds).
5. Download your data as JSON, CSV, Excel, XML or HTML, or send it straight to Google Sheets, Make, Zapier or your AI agent.

![How Website to Markdown works](https://api.apify.com/v2/key-value-stores/wdeAWGooHdIhvFgSY/records/web-to-markdown-workflow.png)

### ⬇️ Input

Add one or more start URLs. With the defaults the Actor converts exactly those pages; everything else is optional.

##### Single pages or a crawl?

At `maxCrawlDepth` 0 (the default) only the start URLs are handled. At 1, the pages they link to on the same host follow, and so on, up to `maxPages` (10 unless changed).

##### Stay in one section

`includeGlobs` and `excludeGlobs` decide which links are followed: `**` matches any depth, `*` a single level. Example: `https://docs.example.com/guide/**`.

##### Chunks for RAG

Switch on `chunks` and set `chunkSize` (200 to 20,000 characters, default 2,000). Each chunk keeps the headings above it, so your retriever knows where the text came from.

| Field | Type | What it does | Default |
|---|---|---|---|
| **Start URLs** (`startUrls`) | array | Pages to convert. | `[{"url":"https://docs.apify.com/academy/` |
| **Crawl depth** (`maxCrawlDepth`) | integer | 0 converts only the start URLs; 1 also the pages they link to on the same host, and so on. | `0` |
| **Maximum pages** (`maxPages`) | integer | Stop after this many pages (each one is a charged event). | `10` |
| **Include globs** (`includeGlobs`) | array | Follow only links matching one of these, e.g. | `[]` |
| **Exclude globs** (`excludeGlobs`) | array | Never follow links matching these, e.g. | `[]` |
| **RAG chunks** (`chunks`) | boolean | Also split each page into chunks with their heading path. | `false` |
| **Chunk size (characters)** (`chunkSize`) | integer | Maximum characters per chunk. | `2000` |
| **Honour robots.txt** (`respectRobotsTxt`) | boolean | Skip pages robots.txt disallows. | `true` |
| **Concurrency** (`maxConcurrency`) | integer | How many pages are fetched in parallel (1 to 20). | `5` |
| **Maximum retries** (`maxRetries`) | integer | Retries per page on network errors, 429 and 5xx. | `2` |

![Website to Markdown input form](https://api.apify.com/v2/key-value-stores/wdeAWGooHdIhvFgSY/records/web-to-markdown-input.png)

#### Input example

```json
{
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
        }
    ],
    "maxCrawlDepth": 1,
    "maxPages": 5,
    "includeGlobs": [
        "https://docs.apify.com/academy/**"
    ]
}
```

### ⬆️ Output

Each result is one row: Markdown plus metadata. The run summary in the key-value store (`OUTPUT`) holds the totals, every failed URL with its error in plain language, and the skipped pages (robots.txt, not HTML, noindex, duplicate after a redirect).

![Website to Markdown output table](https://api.apify.com/v2/key-value-stores/wdeAWGooHdIhvFgSY/records/web-to-markdown-output.png)

#### Output example

One row of the dataset (local crawl of the Apify Academy, 2026-10-16; Markdown and lists shortened):

```json
{
    "url": "https://docs.apify.com/academy/scraping-basics-javascript/devtools-inspecting",
    "requestedUrl": "https://docs.apify.com/academy/scraping-basics-javascript/devtools-inspecting",
    "statusCode": 200,
    "depth": 1,
    "title": "Inspecting web pages with browser DevTools | Academy | Apify Documentation",
    "description": "Lesson about using the browser tools for developers to inspect and manipulate the structure of a website.",
    "language": "en",
    "markdown": "# Inspecting web pages with browser DevTools\n\n**In this lesson we'll use the browser tools for developers to inspect and manipulate the structure of a website.**\n\n---\n\nA browser is the most complete tool for navigating websites. Scrapers are like automated browsers - and sometimes, they actually are automated browsers. ...",
    "headings": [
        {
            "level": 1,
            "text": "Inspecting web pages with browser DevTools"
        },
        {
            "level": 2,
            "text": "Opening DevTools"
        },
        {
            "level": 2,
            "text": "Selecting an element"
        }
    ],
    "links": [
        {
            "url": "https://developer.chrome.com/docs/devtools",
            "text": "Chrome DevTools"
        },
        {
            "url": "https://developer.apple.com/documentation/safari-developer-tools/web-inspector",
            "text": "Web Inspector"
        }
    ],
    "wordCount": 1240,
    "crawledAt": "2026-10-16T06:16:20.296Z"
}
```

#### RAG chunks

With `chunks` on, each row also has a list of pieces. Two of them for the Crawlee Quick Start page (local run with `chunkSize` 1,500, 2026-10-16; 16 in total):

```json
{
    "chunks": [
        {
            "index": 1,
            "headings": [
                "Quick Start"
            ],
            "text": "# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two. To learn in-depth how Crawlee works, read the [Introduction](https://crawlee.dev/js/docs/introduction), which is a comprehensive step-by-step guide for creating your first scraper.",
            "wordCount": 40
        },
        {
            "index": 2,
            "headings": [
                "Quick Start",
                "Choose your crawler​"
            ],
            "text": "## Choose your crawler​\n\nCrawlee comes with three main crawler classes: [`CheerioCrawler`](https://crawlee.dev/js/api/cheerio-crawler/class/CheerioCrawler), [`PuppeteerCrawler`](https://crawlee.dev/js/api/puppeteer-crawler/class/PuppeteerCrawler) and [`PlaywrightCrawler`](https://crawlee.dev/js/api/playwright-crawler/class/PlaywrightCrawler). All classes share the same interface for maximum flexibility when switching between them.",
            "wordCount": 27
        }
    ]
}
```

View the results as a table or JSON in the Output tab and download them as JSON, CSV, Excel, XML, HTML or RSS, all fields or only those you pick.

### 💰 How much does Website to Markdown cost?

Website to Markdown is priced **per event**: **1 USD per 1,000 documents** (0.001 USD per page), with no monthly rental. Failed or skipped inputs (robots.txt, not HTML, noindex, duplicates) cost nothing.

| You get | You pay |
|---|---|
| 100 documents | 0.1 USD |
| 1,000 documents | 1 USD |
| 10,000 documents | 10 USD |
| 100,000 documents | 100 USD |

💡 Apify's free plan gives you 5 USD of credit every month, enough for about **5,000 documents** at no cost. Set a spending limit per run and the Actor stops there.

### ⚖️ Website to Markdown vs. the alternatives

| | Website to Markdown | Browser-based crawlers | Copy and paste |
|---|---|---|---|
| **Menus, footers and cookie banners removed** | ✅ | ✅ usually | ❌ by hand |
| **RAG chunks with heading path** | ✅ | some | ❌ |
| **Headings, links and word count per page** | ✅ | some | ❌ |
| **Speed per page** | ✅ plain HTTP, no browser | slower, a browser per page | minutes |
| **Pages that need JavaScript to show text** | ❌ not supported | ✅ | ✅ |
| **API, schedules and MCP for AI agents** | ✅ | ✅ | ❌ |

### 🔌 Integrations, API and MCP

#### Connect Website to Markdown to your AI agent (MCP)

Give Claude, ChatGPT, Cursor or any MCP client direct access to Website to Markdown through the [Apify MCP server](https://mcp.apify.com). Add this URL to its settings:

```text
https://mcp.apify.com/?actors=bepolabs/web-to-markdown
```

For Claude Desktop, add it under `mcpServers` in `claude_desktop_config.json`:

```json
{
    "mcpServers": {
        "web-to-markdown": {
            "command": "npx",
            "args": [
                "-y",
                "@apify/actors-mcp-server",
                "--actors",
                "bepolabs/web-to-markdown"
            ],
            "env": {
                "APIFY_TOKEN": "<your Apify API token>"
            }
        }
    }
}
```

Then ask: *"Convert the Getting Started pages of docs.example.com to Markdown and summarise how to install the product."*

#### Run it from your code

Python (`pip install apify-client`):

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("bepolabs/web-to-markdown").call(run_input={"startUrls":[{"url":"https://docs.apify.com/academy/web-scraping-for-beginners"}],"maxCrawlDepth":1,"maxPages":5,"includeGlobs":["https://docs.apify.com/academy/**"]})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)
```

JavaScript (`npm install apify-client`):

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('bepolabs/web-to-markdown').call({"startUrls":[{"url":"https://docs.apify.com/academy/web-scraping-for-beginners"}],"maxCrawlDepth":1,"maxPages":5,"includeGlobs":["https://docs.apify.com/academy/**"]});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

#### No-code integrations

Send results to **Google Sheets, Slack, Airtable, HubSpot, Make, Zapier, n8n** and more from the Integrations tab, start runs on a **schedule**, or trigger your own endpoint with a **webhook** when a run finishes. The [Apify API](https://docs.apify.com/api/v2) gives you full control over runs and datasets.

### ❓ FAQ

##### Does it work on pages built with JavaScript?

It reads the HTML the server sends, which covers documentation sites, blogs, news, wikis and most marketing pages. Anything that shows its text only after JavaScript runs (some web apps) comes back nearly empty; the word count makes it easy to spot.

##### How do I convert a whole documentation site?

Give the docs home page as the start URL, set `maxCrawlDepth` to 2 or 3, raise `maxPages`, and add an include glob such as `https://docs.example.com/**` so the crawl stays inside that section.

##### Which chunk size should I use?

1,000 to 2,000 characters suits most embedding models. Chunks break at headings first, then at paragraphs, and never mix two sections.

##### What happens when a page fails?

The other pages still run. The failed URL and the reason (for example HTTP 404 or a timeout) are listed in the run summary, and it is not charged. The run fails only when no page could be converted.

##### Does it follow links to other websites?

No. The crawl stays on the host of each start URL. Links to other sites still appear in the `links` field.

##### Is it legal to use Website to Markdown?

Converting public web pages is generally legal, but the content belongs to its publishers. The Actor honours robots.txt by default, respects `noindex` and `nofollow`, and identifies itself with its own user agent. You are responsible for how you use and republish the text, for copyright and personal data, and for each site's terms; when in doubt, ask the owner.

##### What if something is missing or does not work?

Open an issue on the **Issues** tab; we read each one and reply. Feature requests are welcome too.

### 💬 Your feedback

Tell us what Website to Markdown should add on the **Issues** tab or leave a review; your notes decide what we work on next.

### 🔗 Related Actors

| Actor | What it does |
|---|---|
| 🕰️ [Wayback Machine Scraper](https://apify.com/bepolabs/wayback-snapshots) | The archive history of any URL or domain: first and last seen, every capture, and the page text of old versions. |
| 🔗 [Broken Link Checker](https://apify.com/bepolabs/link-checker) | Every link and image on your pages checked: HTTP status, each redirect hop, final URL and response time. |
| 🩺 [SEO Audit](https://apify.com/bepolabs/seo-audit) | Any page to a 0 to 100 SEO score and grade, with the fixes that matter most first: on-page, links and Core Web Vitals. |
| 🗺️ [Sitemap URL Extractor](https://apify.com/bepolabs/sitemap-urls) | Every URL of a website from its sitemaps, in one deduplicated list with lastmod, changefreq and priority. |
| 🔤 [Image OCR](https://apify.com/bepolabs/image-ocr) | Extract text from images in 29 languages: plain text, clean Markdown and every word with its pixel box and confidence. |
| 📄 [PDF to Text and Markdown](https://apify.com/bepolabs/pdf-to-text) | PDFs to clean text and Markdown, per page, per document or as RAG chunks, with columns in reading order and headers and footers removed. |
| 🌐 [Domain RDAP and DNS](https://apify.com/bepolabs/domain-rdap) | Domains to registrar, creation and expiry dates, age, status, nameservers, DNSSEC and DNS records from official RDAP, in bulk. |
| ⚡ [Bulk PageSpeed Insights](https://apify.com/bepolabs/pagespeed) | Lighthouse scores, lab metrics and real-user Core Web Vitals for any list of URLs, on mobile, desktop or both. |

# Changelog

This Actor's version history is a separate document: https://apify.com/bepolabs/web-to-markdown/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

Pages to convert. With a crawl depth above 0, links on the same host are followed too.

## `maxCrawlDepth` (type: `integer`):

0 converts only the start URLs; 1 also the pages they link to on the same host, and so on.

## `maxPages` (type: `integer`):

Stop after this many pages (each one is a charged event).

## `includeGlobs` (type: `array`):

Follow only links matching one of these, e.g. https://example.com/docs/\*\*. \*\* matches anything, \* anything but /.

## `excludeGlobs` (type: `array`):

Never follow links matching these, e.g. **/tag/**.

## `chunks` (type: `boolean`):

Also split each page into chunks with their heading path.

## `chunkSize` (type: `integer`):

Maximum characters per chunk.

## `respectRobotsTxt` (type: `boolean`):

Skip pages robots.txt disallows.

## `maxConcurrency` (type: `integer`):

How many pages are fetched in parallel (1 to 20). Lower it for slow or rate-limited websites.

## `maxRetries` (type: `integer`):

Retries per page on network errors, 429 and 5xx.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ],
  "maxCrawlDepth": 0,
  "maxPages": 10,
  "includeGlobs": [],
  "excludeGlobs": [],
  "chunks": false,
  "chunkSize": 2000,
  "respectRobotsTxt": true,
  "maxConcurrency": 5,
  "maxRetries": 2
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset items, one row per result, in the overview table view.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("bepolabs/web-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }] }

# Run the Actor and wait for it to finish
run = client.actor("bepolabs/web-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ]
}' |
apify call bepolabs/web-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,bepolabs/web-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/V5TzHiLF0FOZKjODo/builds/imZ5YI20dfYUagJDt/openapi.json