# URL to Markdown: Website to Markdown for LLM & RAG (`johto.data/url-to-markdown`) Actor

Convert any URL or whole website to clean Markdown for LLMs, RAG, vector databases and AI agents. HTML to Markdown with menus, ads and footers removed. Keeps headings, lists, tables and links. Token estimate per page. $0.80 per 1,000 pages.

- **URL**: https://apify.com/johto.data/url-to-markdown.md
- **Developed by:** [Johto Data](https://apify.com/johto.data) (community)
- **Categories:** AI, Developer tools, Agents
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.80 / 1,000 page converteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## URL to Markdown for LLM and RAG

Convert web pages or a site section to Markdown. Main-content extraction removes navigation where detected and preserves headings, paragraphs, lists, tables, code blocks, links and images.

### Pricing

**$0.80 per 1,000 pages converted.** No tool subscription or start fee. Pages skipped by robots.txt or that fail to load are not charged.

Example: 2,000 converted pages cost $1.60.

### Quick start

1. Paste page URLs into `startUrls`.
2. Use `maxDepth: 0` for those URLs only. Set a higher depth to follow links, and set `maxPages` to cap the run.
3. Press **Start**. Export JSON, CSV or Excel, or retrieve the dataset through the API.

### Example input

Convert one page:

```json
{ "startUrls": [{ "url": "https://en.wikipedia.org/wiki/Markdown" }] }
```

Crawl a docs section:

```json
{
  "startUrls": [{ "url": "https://crawlee.dev/js/docs/quick-start" }],
  "maxDepth": 2,
  "maxPages": 200,
  "includeUrlGlobs": ["https://crawlee.dev/js/docs/**"]
}
```

### Output

- `markdown`: clean GitHub-flavored Markdown (tables and fenced code blocks)
- `title`, `description`, `language`, `canonicalUrl`
- `wordCount` and `tokenEstimate` (characters divided by 4, a common rough estimate for LLM tokens)
- `statusCode`, `depth`, `crawledAt`
- Optional `html`: the cleaned HTML used for the conversion

```json
{
  "url": "https://crawlee.dev/js/docs/introduction/setting-up",
  "title": "Setting up | Crawlee for JavaScript · Build reliable crawlers. Fast.",
  "language": "en",
  "depth": 1,
  "wordCount": 473,
  "tokenEstimate": 886,
  "markdown": "# Setting up\n\n...",
  "crawledAt": "2026-11-08T14:50:00.000Z"
}
```

Export to JSON, CSV or Excel, or pull results by API into LangChain, LlamaIndex, Pinecone or another pipeline.

### Input options

| Field | Default | What it does |
|---|---|---|
| `startUrls` | required | Pages to convert |
| `maxDepth` | 0 | 0 converts only the given URLs. 1 or more also follows same-site links |
| `maxPages` | 100 | Hard cap on pages for the run |
| `includeUrlGlobs` | none | Only follow links matching these globs, for example `https://example.com/docs/**` |
| `onlyMainContent` | true | Remove navigation, headers, footers and sidebars |
| `includeLinks` | true | Keep hyperlinks as Markdown links |
| `includeHtml` | false | Also return the cleaned HTML |
| `respectRobotsTxt` | true | Skip pages disallowed by robots.txt |
| `proxyConfiguration` | none | Optional Apify Proxy settings |

### API and integrations

Call `johto.data/url-to-markdown` through the [Apify API](https://docs.apify.com/api/v2) or [Apify MCP server](https://mcp.apify.com). Use the input above as the request body. Retrieve the resulting Markdown from the dataset.

### Use cases

- Feed documentation, help centers and blogs into a RAG index or vector database
- Give an AI agent readable page content instead of raw HTML
- Build training or evaluation sets from public pages
- Archive articles as Markdown notes

### Limits

The Actor uses HTTP requests (no browser), which keeps the price low. Pages that only render their text with JavaScript may return little content. Sites with bot protection may block requests. A proxy is not permission to bypass a restriction.

**Does it run JavaScript?** No. It reads the HTML the server sends. JavaScript-only content may be missing.

**Can I use it from an AI agent?** Yes. Call it through the Apify API or the Apify MCP server and read the dataset.

**How is the token estimate computed?** Markdown length in characters divided by 4. Use it for budgeting, not exact counts.

### Use with AI agents and automation

**Apify MCP server (Claude, Cursor, ChatGPT, any MCP client).** Add `https://mcp.apify.com?tools=johto.data/url-to-markdown` as a remote MCP server. The agent gets one tool that takes the input below and returns one row per page with `markdown`, `title` and a token estimate.

**API, one call.** Runs and returns the dataset in the same request:

```bash
curl -X POST "https://api.apify.com/v2/acts/johto.data~url-to-markdown/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://docs.apify.com/academy/web-scraping-for-beginners"}],"maxDepth":0,"maxPages":10}'
```

**LangChain (Python).**

```python
from langchain_apify import ApifyActorsTool
tool = ApifyActorsTool("johto.data/url-to-markdown")
```

**n8n, Make, Zapier.** Use the Apify app or node, choose **Run Actor**, enter `johto.data/url-to-markdown` and paste the input JSON. Then map dataset fields into Sheets, a CRM or a database.

### FAQ

**How does the price compare?** $0.80 per 1,000 pages, no start fee. Apify's own URL to Markdown lists $3 per 1,000 pages over plain HTTP on the free plan (checked Nov 8, 2026).

**Does it render JavaScript?** No. It uses plain HTTP, which is fast and cheap. Pages that need a browser to show content return less text.

**Can it crawl a whole docs site?** Yes. Set `maxDepth` above 0, cap it with `maxPages` and narrow it with `includeUrlGlobs`.

**How long does a run take?** The prefilled input finishes in about 3 seconds.

### Responsible use

Convert only content you own, have permission to process, or may use under an applicable license. Public readability does not grant permission for commercial redistribution, model training or removal of attribution. Check each source's terms, robots.txt and content license. Preserve required attribution and license notices in downstream use.

The tool respects robots.txt by default. Do not disable that setting to bypass a source's restriction. Do not crawl behind logins or bypass access controls. Proxy configuration is for permitted access, not evading a source's prohibition.

Page text can contain personal data. You are responsible for a lawful basis and applicable privacy duties before collecting, storing or reusing it. Markdown conversion does not remove copyright, privacy or data-protection obligations.

### Support

Open an issue in the **Issues** tab with the input URL and a description of the problem. Do not include API tokens or private data. Built and maintained by [Johto Data](https://johtodata.com).

# Actor input Schema

## `startUrls` (type: `array`):

Pages to convert. Turn on crawling below to also follow links.

## `maxDepth` (type: `integer`):

0 converts only the given URLs. 1 or more also follows links on the same site, up to this many clicks away.

## `maxPages` (type: `integer`):

Hard cap on pages for the run. You pay per page converted.

## `includeUrlGlobs` (type: `array`):

Optional glob patterns, for example https://example.com/docs/\*\*

## `onlyMainContent` (type: `boolean`):

Remove navigation, headers, footers, cookie banners and sidebars.

## `includeLinks` (type: `boolean`):

Keep hyperlinks as Markdown links. Turn off for plain text style output.

## `includeHtml` (type: `boolean`):

Add the cleaned HTML next to the Markdown.

## `respectRobotsTxt` (type: `boolean`):

Skip pages that the site's robots.txt disallows.

## `proxyConfiguration` (type: `object`):

Optional.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ],
  "maxDepth": 0,
  "maxPages": 100,
  "onlyMainContent": true,
  "includeLinks": true,
  "includeHtml": false,
  "respectRobotsTxt": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
        }
    ],
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("johto.data/url-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }],
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("johto.data/url-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call johto.data/url-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,johto.data/url-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/jQQUCBbKu1CdpydRg/builds/1SdcajipsvfOpUWim/openapi.json