# Website to Markdown for AI, LLMs & RAG (`veedel_digital/website-markdown`) Actor

Crawl a website or a list of URLs and get clean Markdown of the main content for ChatGPT, LLMs, RAG and vector databases. Removes menus, footers and cookie banners, keeps tables and links, and builds one llms-full.md file. No browser, no proxies, pay per page.

- **URL**: https://apify.com/veedel_digital/website-markdown.md
- **Developed by:** [Tim Tonat](https://apify.com/veedel_digital) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does Website to Markdown for AI, LLMs & RAG do?

This Actor **crawls a website and turns every page into clean Markdown**, ready for ChatGPT, Claude, custom GPTs, RAG pipelines and vector databases. Enter a homepage (or a list of URLs) and get **one row per page** with the main content as Markdown, plus **one combined `llms-full.md` file** with the whole site.

Menus, headers, footers, forms, language switchers and cookie banners are removed automatically. Headings, lists, **tables**, links and images are kept, and relative links become absolute URLs. It reads the HTML directly, **without a browser and without proxies**, so runs are fast and cheap.

### Why use this website to Markdown converter?

- **Build a knowledge base for your chatbot**: feed your docs, help center or product pages into a custom GPT, Claude Project or RAG app
- **Create `llms-full.txt` for your website**: the combined file lists every page with its title and source URL
- **Clean input for embeddings**: no navigation noise repeated on every page, so vector search finds real content
- **Stay focused with URL patterns**: crawl only `/docs/**` or exclude `/tag/**`
- **Pay per page**: you control the cost with *Max pages*
- Run it from the Apify API, Make, Zapier, n8n or LangChain, or schedule it to keep your knowledge base fresh

### What data does it return?

| Field | Description |
|---|---|
| `url` | Final URL of the page |
| `title`, `description`, `lang` | Page title, meta description and language |
| `markdown` | Main content as Markdown |
| `wordCount` | Words in the main content |
| `crawlDepth`, `foundOn` | Clicks from the start URL and the page that linked to it |
| `truncated` | True if the Markdown was cut at 2 million characters |
| `error` | Reason if the page could not be loaded |

The key-value store record **`llms-full.md`** contains all pages in one file, each starting with its title and source URL. It is limited to 8 MB; the dataset always contains every page in full.

### How to convert a website to Markdown

1. Click **Try for free**.
2. Enter your homepage in **Start URLs**.
3. Set **Max pages** (start with 20 to check the result).
4. Optional: limit the crawl with **Include URLs**, e.g. `https://example.com/docs/**`.
5. Click **Start**.
6. Download the dataset as JSON, CSV or Excel, or open **llms-full.md** in the Storage tab.

### How much does it cost?

This Actor uses **pay per event**: a fixed price **per converted page**, and **platform usage is included**. The current price is shown in the **Pricing** tab. Pages that return an error or can't be reached are not charged. With *Max pages* you always know the maximum cost in advance.

### Input

See the **Input** tab for all options.

```json
{
    "startUrls": [{ "url": "https://example.com/docs/" }],
    "crawlSite": true,
    "maxPages": 100,
    "includeUrlGlobs": ["https://example.com/docs/**"],
    "excludeUrlGlobs": ["**/changelog/**"],
    "removeSelectors": ".newsletter-box, #related-posts"
}
```

### Output

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

````json
{
    "url": "https://example.com/docs/getting-started/",
    "statusCode": 200,
    "crawlDepth": 1,
    "foundOn": "https://example.com/docs/",
    "title": "Getting started | Example Docs",
    "description": "Install Example and run your first project in five minutes.",
    "lang": "en",
    "wordCount": 640,
    "truncated": false,
    "markdown": "# Getting started\n\nInstall the CLI:\n\n```\nnpm i -g example\n```\n\n| Plan | Price |\n| --- | --- |\n| Free | $0 |",
    "error": null
}
````

### Related Actors

- [SEO Site Audit & Broken Link Checker](https://apify.com/veedel_digital/seo-audit): crawl a whole website and find on-page SEO issues and broken pages
- [Structured Data, Schema.org & Open Graph Extractor](https://apify.com/veedel_digital/structured-data): extract JSON-LD, schema.org types and Open Graph tags from many pages
- [PDF to Text with OCR](https://apify.com/veedel_digital/pdf-to-text): your knowledge base also has PDFs? Extract their text, scanned pages included
- [PageSpeed Insights & Core Web Vitals Bulk Audit](https://apify.com/veedel_digital/pagespeed-audit): Lighthouse scores and real-user Core Web Vitals for many URLs

### FAQ

#### Does it work with JavaScript-heavy sites?

The Actor reads the HTML the server delivers. Most websites, docs and blogs deliver their content this way. Single-page apps that render everything in the browser return little or no content; for those, use a browser-based crawler.

#### How does it find the main content?

It uses the page's `<main>` element (or a single `<article>`) when there is one. Otherwise it takes the whole page and removes header, footer, navigation, forms and cookie banners. If a site has extra boxes you don't want, add their CSS selectors in **Remove elements**.

#### Will it overload my server?

No. It follows `robots.txt` by default and uses at most 10 parallel requests.

#### Something does not work

Please open an issue in the **Issues** tab with the URL and the run ID. You can also start the Actor programmatically from the **API** tab.

# Actor input Schema

## `startUrls` (type: `array`):

Homepage or pages to convert. With "Crawl whole website" enabled, internal links on the same hostname are followed.

## `crawlSite` (type: `boolean`):

Follow internal links on the same hostname. Disable to convert only the URLs you entered.

## `maxPages` (type: `integer`):

Maximum number of pages to convert (each converted page is charged).

## `includeUrlGlobs` (type: `array`):

Only follow links matching one of these patterns, e.g. https://example.com/docs/\*\*. Leave empty to follow all internal links.

## `excludeUrlGlobs` (type: `array`):

Never follow links matching these patterns, e.g. **/tag/** or **/login**.

## `removeSelectors` (type: `string`):

Extra elements to remove before conversion, e.g. .newsletter, #related-posts. Menus, headers, footers, forms and cookie banners are removed automatically.

## `respectRobotsTxt` (type: `boolean`):

Skip pages disallowed by the site's robots.txt.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy"
    }
  ],
  "crawlSite": true,
  "maxPages": 100,
  "respectRobotsTxt": true
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset with the Markdown of every page

## `llmsFull` (type: `string`):

All pages in one Markdown file

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("veedel_digital/website-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://docs.apify.com/academy" }] }

# Run the Actor and wait for it to finish
run = client.actor("veedel_digital/website-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy"
    }
  ]
}' |
apify call veedel_digital/website-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,veedel_digital/website-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/hcSxKb6pkDB3wXpz1/builds/aXTKWlbj2vA7riAPJ/openapi.json