URL to Markdown for LLMs & RAG – Pages, PDFs, Sitemaps & Small Crawls

What does URL to Markdown do?

URL to Markdown is a web page to Markdown converter and HTML-to-Markdown API for LLMs, RAG pipelines and AI agents. Give it URLs – web pages, PDFs, a sitemap or a small website section – and it returns the main text as clean Markdown, plain text or HTML, with metadata and optional token-sized chunks.

Turn public web pages and PDFs into clean, LLM-ready Markdown (or plain text / cleaned HTML) – main content only (no menus, ads or cookie text), with title, author, publish date, language, description and canonical URL. Optionally follow links or a sitemap to convert a whole docs section, and split the text into token-sized chunks with overlap for RAG / vector databases.

Pages are fetched with fast HTTP requests first; a headless browser is used only when a page needs JavaScript – so most pages convert in about a second, and you only pay for pages that were converted. Errors, blocked pages, robots.txt-disallowed URLs and duplicate pages are free.

Why use it

• Main-content extraction (article mode) or full-page Markdown – your choice; fine-tune with CSS selectors to keep or remove elements
• Output formats: markdown, text, html – one or several per page
• PDF to text/Markdown (up to N pages) – the same price as a web page
• Small crawls: follow links up to a depth, or start from the site's sitemap.xml, filter with include/exclude URL globs, limit pages per start URL; stays on the same host; duplicate pages (same final URL or same content) are skipped and not charged
• JavaScript pages: auto browser fallback, optional wait for selector and extra wait time
• Keeps links (absolute URLs), tables and optionally images; optional list of internal/external links per page
• Chunking for RAG: chunk size and overlap in approximate tokens, split on paragraph boundaries
• Respects robots.txt by default and identifies itself with an honest User-Agent; custom headers supported
• Parallel fetching (default 8), retries with backoff on 429/5xx, optional Apify Proxy
• Works as a tool for AI agents via the Apify API and MCP

How to use URL to Markdown

1. Open the Actor in Apify Console (click Try for free on this page) and sign in or create a free Apify account.
2. Paste the pages or PDF links into URLs. To convert a whole section, set Crawl depth above 0 or turn on Use sitemap, and limit it with Max pages per start URL (crawl) and Include URL patterns / Exclude URL patterns.
3. Pick Output formats (markdown, text, html) and an Extraction mode (auto, article for main content only, or full).
4. Optional for RAG: set Chunk size (approx. tokens) and Chunk overlap (approx. tokens). For JavaScript-heavy sites, use JavaScript rendering and Wait for selector (browser).
5. Click Start. When the run finishes, open the Output tab and download the results as JSON, CSV, Excel or HTML.
6. To repeat it, create a Schedule, or call the Actor from your app via the Apify API.

Input examples

Convert a list of pages:

{
  "urls": ["https://react.dev/learn", "https://arxiv.org/pdf/1706.03762"],
  "outputFormats": ["markdown"],
  "chunkSizeTokens": 800,
  "chunkOverlapTokens": 80
}

Convert a docs section from its sitemap:

{
  "urls": ["https://docs.apify.com/"],
  "useSitemap": true,
  "includeUrlGlobs": ["https://docs.apify.com/academy/*"],
  "maxPagesPerStartUrl": 50
}

Follow links two levels deep and drop page furniture:

{
  "urls": ["https://books.toscrape.com/"],
  "maxCrawlDepth": 2,
  "maxPagesPerStartUrl": 100,
  "excludeUrlGlobs": ["*/category/*"],
  "removeElementsCssSelector": "nav, footer, .sidebar"
}

Output example

{
  "url": "https://react.dev/learn",
  "finalUrl": "https://react.dev/learn",
  "statusCode": 200,
  "title": "Quick Start – React",
  "language": "en",
  "publishedAt": null,
  "markdown": "## Quick Start\n\nWelcome to the React documentation! ...",
  "characters": 18342,
  "words": 2711,
  "approxTokens": 4585,
  "fetchedWith": "http",
  "crawlDepth": null,
  "chunks": [{ "index": 0, "text": "## Quick Start ...", "approxTokens": 790 }]
}

fetchedWith is http, browser or pdf. With includeLinkList you also get links.internal / links.external. Failed URLs come back as rows with error (and blocked: true for 401/403/429/bot walls) and are not charged.

How much does it cost?

URL to Markdown uses pay-per-event pricing: $0.002 per converted page ($2 per 1,000 pages). Web pages and PDFs cost the same, and you don't pay separately for compute.

Pay per converted page (web page or PDF – same price). Failed, blocked, robots.txt-disallowed and duplicate URLs are free. Set a maximum cost per run and the Actor stops cleanly when it is reached – useful for crawls. The only other charge is Apify's standard Actor start event ($0.00005 per run, per GB of memory).

Apify's free plan includes $5 of platform credit every month, which is enough to try the Actor on real data before you pay anything.

Use with AI agents (MCP)

URL to Markdown works as a ready-made tool for Claude, ChatGPT, Cursor, VS Code and any other MCP client through the official Apify MCP server. The agent gets one tool, cleandata-labs--url-to-markdown, with the input fields described on this page, and you pay the normal pay-per-event price from your Apify account (a free account works; it includes $5 of monthly credit).

MCP server URL with only this Actor:

https://mcp.apify.com?tools=cleandata-labs/url-to-markdown

On first connection your client opens the Apify sign-in (OAuth). To use an API token instead, send the header Authorization: Bearer <YOUR_APIFY_TOKEN>.

Claude (claude.ai and Claude Desktop): Settings → Connectors → Add custom connector. Name: URL to Markdown, Remote MCP server URL: the URL above.

Claude Desktop with a local config file (claude_desktop_config.json, needs Node.js 18+):

{
  "mcpServers": {
    "url-to-markdown": {
      "command": "npx",
      "args": ["-y", "@apify/actors-mcp-server", "--tools", "cleandata-labs/url-to-markdown"],
      "env": { "APIFY_TOKEN": "<YOUR_APIFY_TOKEN>" }
    }
  }
}

Cursor (~/.cursor/mcp.json, or .cursor/mcp.json in a project):

{
  "mcpServers": {
    "url-to-markdown": {
      "url": "https://mcp.apify.com?tools=cleandata-labs/url-to-markdown"
    }
  }
}

Claude Code:

$claude mcp add --transport http url-to-markdown "https://mcp.apify.com?tools=cleandata-labs/url-to-markdown"

VS Code (.vscode/mcp.json):

{
  "servers": {
    "url-to-markdown": { "type": "http", "url": "https://mcp.apify.com?tools=cleandata-labs/url-to-markdown" }
  }
}

Already connected to https://mcp.apify.com with the default tools? Just ask your agent to use cleandata-labs/url-to-markdown; it can run it through call-actor. To load several Actors in one connection, separate them with commas, e.g. https://mcp.apify.com?tools=cleandata-labs/google-trends-explorer,cleandata-labs/url-to-markdown.

Tips for agents

• Smallest useful input: {"urls": ["https://docs.apify.com/platform/actors"], "maxCharacters": 20000} (one page as clean Markdown, cut to ~5k tokens).
• A single page usually takes 2-5 seconds; small crawls (10-20 pages) well under a minute, so results are usually ready within the MCP call's default 30-second wait. The call returns a dataset ID; the agent reads the rows with get-dataset-items.
• Cost: $0.002 per converted page or PDF. Blocked, 4xx, robots.txt-disallowed and duplicate URLs are free.
• Each row has approxTokens, so the agent can decide how much to read. Use maxCharacters to keep long pages small.
• Always pass your own urls: the default list is only a demo.
• For a whole docs section use maxCrawlDepth: 1 or useSitemap: true with maxPagesPerStartUrl (e.g. 10) and includeUrlGlobs.
• For RAG set chunkSizeTokens (e.g. 800) to get ready-made chunks.

Example prompts

1. "Read https://docs.apify.com/platform/actors as Markdown and summarize it in 5 bullet points."
2. "Convert up to 10 pages under https://crawlee.dev/python/docs/ into Markdown chunks of about 800 tokens for my RAG index, and list the page titles."
3. "Extract the text of this PDF:

Integrations

URL to Markdown works with everything that connects to Apify:

• Make, Zapier and n8n – start a run and process the results in your no-code workflows.
• Google Sheets – export the dataset to a spreadsheet, or keep a sheet updated with a scheduled run.
• Webhooks – get a notification (Slack, email, your own endpoint) when a run finishes or fails.
• Apify API and clients – start runs and download results from your code with the REST API, the Python client (apify-client) or the JavaScript client (apify-client).
• AI agents – use it as a tool through the Apify MCP server or the LangChain / LlamaIndex integrations.

Python example:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("cleandata-labs/url-to-markdown").call(run_input={"urls": ["https://docs.apify.com/platform/actors"], "chunkSizeTokens": 800})
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

Notes & limitations

• Pages behind logins, paywalls or bot protection are not bypassed; they come back as error rows (blocked: true).
• Crawling is intentionally small-scale (same host, page limits). For crawling entire large websites use a dedicated crawler.
• PDFs: text is extracted (no OCR for scanned PDFs); tables come out as plain text.
• "Tokens" are approximated as characters ÷ 4; exact counts depend on your model's tokenizer.
• You are responsible for complying with each website's terms and copyright when storing or reusing content.

FAQ

Is it legal to convert web pages to Markdown? The Actor only fetches publicly accessible pages, respects robots.txt by default and does not bypass logins or bot protection. Web content can be protected by copyright and site terms, and pages may contain personal data, so you are responsible for making sure your use (storing, training, republishing) complies with the website's terms and applicable law. Ask a lawyer if you are unsure.

Can I use it with integrations, the API or LLM tools? Yes. Call it from Python or JavaScript with the Apify API, connect it to Make, Zapier, n8n, Google Sheets or webhooks, or give it to AI agents through the Apify MCP server or LangChain / LlamaIndex. The Markdown and chunks fields are ready to send to a vector database.

Does it work with JavaScript-rendered pages? Yes. With JavaScript rendering set to auto, pages that return too little text over plain HTTP are opened in a headless browser. Use always for single-page apps and Wait for selector (browser) if content loads late.

Can it convert PDFs? Yes – text-based PDFs, up to Max PDF pages per file, at the same price as a web page. Scanned PDFs (images only) are not OCR'd.

Why was a URL not charged? Failed, blocked (401/403/429 or bot walls), robots.txt-disallowed and duplicate pages are returned as error rows or skipped, and are free.