# Website to Markdown: Fast Crawler for LLM & RAG (No Browser) (`arnaud437/website-to-markdown`) Actor

Turn web pages into clean Markdown for LLMs, RAG and AI agents: main content only, menus and footers removed, can follow site links. $1 per 1,000 pages.

- **URL**: https://apify.com/arnaud437/website-to-markdown.md
- **Developed by:** [Open Data Tools](https://apify.com/arnaud437) (community)
- **Categories:** AI, Developer tools, Automation
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 page converteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown: Fast Crawler for LLM & RAG (No Browser)

Give **page links**, get **clean Markdown** ready for an LLM, a RAG index or an AI agent: headings, paragraphs, lists, links, tables and code, **without the menus, headers, footers, sidebars and cookie banners**. One page, or a whole documentation: the Actor can follow the links of a site under the same path. It reads pages directly, without a browser, so it is fast and cheap: **$1 per 1,000 pages**.

### What you get

One row per page:

- **Markdown** of the main content (or of the whole page if you prefer).
- **Title**, **description**, **language** and **canonical link**.
- **Dates** the page gives: published and modified, in ISO 8601.
- **Counts**: words, links and images in the Markdown.
- **Depth** of the page from the link you gave (0 for the page itself).

### Use cases

- **RAG and chatbots**: feed a documentation, a help center or a blog into a vector database.
- **AI agents**: one call to read a page as text an LLM understands.
- **Content audits**: word counts and structure of many pages at once.
- **Archives and monitoring**: keep the text of pages that matter to you.

### How to use

- **Without changing anything**: the Apify documentation page about Actors.
- **Pages**: paste links or bare domains, one per line, up to 10,000 per run.
- **Pages per link**: 1 converts the page alone. More follows its links on the same site, under the same path: from `https://example.com/docs` the Actor reads `/docs/...` pages only, up to the number you choose.
- **Main content only**, **Keep links**, **Keep images**: switches for shorter or fuller text.

### Input example

```json
{
  "urls": ["https://docs.apify.com/platform/actors"],
  "maxPagesPerUrl": 5,
  "onlyMainContent": true,
  "includeLinks": true,
  "includeImages": false
}
```

### Output example

```json
{
  "recordType": "page",
  "id": "https://docs.apify.com/actors",
  "url": "https://docs.apify.com/actors",
  "startUrl": "https://docs.apify.com/platform/actors",
  "depth": 0,
  "title": "Actors | Platform | Apify Documentation",
  "description": "Learn how to develop, run and share serverless cloud programs.",
  "lang": "en",
  "canonicalUrl": "https://docs.apify.com/actors",
  "publishedAt": null,
  "modifiedAt": null,
  "markdown": "# Actors\n\nActors are serverless cloud programs that take a structured JSON input, perform a task…",
  "wordCount": 360,
  "linkCount": 16,
  "imageCount": 0,
  "status": "ok",
  "sourceName": "Website docs.apify.com",
  "sourceUrl": "https://docs.apify.com/actors",
  "licence": "Text of a public page, as its owner publishes it; check the site terms before reuse",
  "sourceUpdatedAt": null,
  "actorVersion": "0.1.1",
  "retrievedAt": "2026-11-17T21:00:00.000Z"
}
```

### Pricing

Pay per event: **$1.00 per 1,000 pages converted** (`page-converted`, $0.001 each), plus Apify's start fee of $0.00005 per run. Pages that do not answer, are missing or are not web pages give a **free** error row.

| Pages   | Price |
| ------- | ----- |
| 100     | $0.10 |
| 1,000   | $1    |
| 100,000 | $100  |

### Data source

The pages you give, read directly as their server sends them. The Actor introduces itself as `OpenDataToolsBot`, follows robots.txt (RFC 9309), waits between two pages of the same site, and never logs in, solves no CAPTCHA and uses no proxy to get round a block.

**Limit**: pages that build their text with JavaScript after loading (some single-page apps) give little or no Markdown, since no browser runs here. Most sites, documentations, blogs and news pages send their text directly.

### Is it legal?

The Actor reads public pages, as each site's robots.txt allows. The text stays its owner's: check the site's terms before you republish it or use it to train a model.

### FAQ

- **Why so few words on a page?** The page may build its text with JavaScript, or the main part may be small. Try "Main content only" off.
- **Why did the crawl stop early?** It reads links under the path you gave, on the same site, and skips pages that robots.txt closes, files (PDF, images) and pages that fail.
- **Can I get shorter text for an AI model?** Turn "Keep links" off: links keep their text only.

### Related Actors

- [AI Model Prices](https://apify.com/arnaud437/ai-model-prices): Compare 400+ LLM Prices and Features
- [llms.txt Generator](https://apify.com/arnaud437/llms-txt-generator): AI-Ready Site Map for Any Website
- [AI Readiness Checker](https://apify.com/arnaud437/ai-readiness-checker): GPTBot, ClaudeBot, llms.txt & AI SEO
- [Structured Data Extractor](https://apify.com/arnaud437/structured-data-extractor): Schema.org, JSON-LD & Open Graph
- [Sitemap Extractor](https://apify.com/arnaud437/sitemap-extractor): All URLs of a Site, Lastmod & Priority

Every Actor here also works as a tool for AI agents (Claude, ChatGPT, Cursor…) through [Apify’s MCP server](https://mcp.apify.com).

### Changelog

See [CHANGELOG.md](CHANGELOG.md).

### En français

#### Pages web en Markdown propre, pour les IA

Donnez des liens de pages, obtenez leur texte en Markdown (titres, listes, liens, tableaux, code) sans les menus, en-têtes, pieds de page et bandeaux cookies. Le robot peut suivre les liens d'un site sous le même chemin (toute une documentation, par exemple).

#### Prix

1 $ les 1 000 pages. Les lignes d'erreur sont gratuites.

#### Source et légalité

Les pages publiques elles-mêmes, lues dans le respect de leur robots.txt, sans contournement. Le texte reste à son auteur : vérifiez les conditions du site avant de le republier.

# Changelog

This Actor's version history is a separate document: https://apify.com/arnaud437/website-to-markdown/changelog.md

# Actor input Schema

## `urls` (type: `array`):

One page link per line (https://www.example.com/blog/post) or a domain (example.com). Up to 10,000 per run; each link is read once.

## `maxPagesPerUrl` (type: `integer`):

1 converts the page alone. More follows its links on the same site, under the same path (for example everything under /docs), up to this number of pages. Each page is charged.

## `onlyMainContent` (type: `boolean`):

Keep the article or main part of the page and leave out menus, headers, footers, sidebars, sharing buttons and cookie banners. Turn off to keep the whole page.

## `includeLinks` (type: `boolean`):

Write links as [text](url). Turn off to keep their text only, which makes shorter text for an AI model.

## `includeImages` (type: `boolean`):

Write images as ![description](url).

## Actor input object example

```json
{
  "urls": [
    "https://docs.apify.com/platform/actors"
  ],
  "maxPagesPerUrl": 1,
  "onlyMainContent": true,
  "includeLinks": true,
  "includeImages": false
}
```

# Actor output Schema

## `results` (type: `string`):

One row per page: its Markdown, title, description, language and dates.

## `summary` (type: `string`):

Pages converted, error rows, and dates of the run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://docs.apify.com/platform/actors"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("arnaud437/website-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://docs.apify.com/platform/actors"] }

# Run the Actor and wait for it to finish
run = client.actor("arnaud437/website-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://docs.apify.com/platform/actors"
  ]
}' |
apify call arnaud437/website-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,arnaud437/website-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/6wTgFCuwrKcc1PoaW/builds/5Iho1nUpJmz0vtRRk/openapi.json
