URL to Markdown for LLM and RAG

Convert web pages or a site section to Markdown. Main-content extraction removes navigation where detected and preserves headings, paragraphs, lists, tables, code blocks, links and images.

Pricing

$0.80 per 1,000 pages converted. No tool subscription or start fee. Pages skipped by robots.txt or that fail to load are not charged.

Example: 2,000 converted pages cost $1.60.

Quick start

1. Paste page URLs into startUrls.
2. Use maxDepth: 0 for those URLs only. Set a higher depth to follow links, and set maxPages to cap the run.
3. Press Start. Export JSON, CSV or Excel, or retrieve the dataset through the API.

Example input

Convert one page:

{ "startUrls": [{ "url": "https://en.wikipedia.org/wiki/Markdown" }] }

Crawl a docs section:

{
  "startUrls": [{ "url": "https://crawlee.dev/js/docs/quick-start" }],
  "maxDepth": 2,
  "maxPages": 200,
  "includeUrlGlobs": ["https://crawlee.dev/js/docs/**"]
}

Output

• markdown: clean GitHub-flavored Markdown (tables and fenced code blocks)
• title, description, language, canonicalUrl
• wordCount and tokenEstimate (characters divided by 4, a common rough estimate for LLM tokens)
• statusCode, depth, crawledAt
• Optional html: the cleaned HTML used for the conversion

{
  "url": "https://crawlee.dev/js/docs/introduction/setting-up",
  "title": "Setting up | Crawlee for JavaScript · Build reliable crawlers. Fast.",
  "language": "en",
  "depth": 1,
  "wordCount": 473,
  "tokenEstimate": 886,
  "markdown": "# Setting up\n\n...",
  "crawledAt": "2026-11-08T14:50:00.000Z"
}

Export to JSON, CSV or Excel, or pull results by API into LangChain, LlamaIndex, Pinecone or another pipeline.

Input options

Field	Default	What it does
startUrls	required	Pages to convert
maxDepth	0	0 converts only the given URLs. 1 or more also follows same-site links
maxPages	100	Hard cap on pages for the run
includeUrlGlobs	none	Only follow links matching these globs, for example https://example.com/docs/**
onlyMainContent	true	Remove navigation, headers, footers and sidebars
includeLinks	true	Keep hyperlinks as Markdown links
includeHtml	false	Also return the cleaned HTML
respectRobotsTxt	true	Skip pages disallowed by robots.txt
proxyConfiguration	none	Optional Apify Proxy settings

API and integrations

Call johto.data/url-to-markdown through the Apify API or Apify MCP server. Use the input above as the request body. Retrieve the resulting Markdown from the dataset.

Use cases

• Feed documentation, help centers and blogs into a RAG index or vector database
• Give an AI agent readable page content instead of raw HTML
• Build training or evaluation sets from public pages
• Archive articles as Markdown notes

Limits

The Actor uses HTTP requests (no browser), which keeps the price low. Pages that only render their text with JavaScript may return little content. Sites with bot protection may block requests. A proxy is not permission to bypass a restriction.

Does it run JavaScript? No. It reads the HTML the server sends. JavaScript-only content may be missing.

Can I use it from an AI agent? Yes. Call it through the Apify API or the Apify MCP server and read the dataset.

How is the token estimate computed? Markdown length in characters divided by 4. Use it for budgeting, not exact counts.

Use with AI agents and automation

Apify MCP server (Claude, Cursor, ChatGPT, any MCP client). Add https://mcp.apify.com?tools=johto.data/url-to-markdown as a remote MCP server. The agent gets one tool that takes the input below and returns one row per page with markdown, title and a token estimate.

API, one call. Runs and returns the dataset in the same request:

curl -X POST "https://api.apify.com/v2/acts/johto.data~url-to-markdown/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://docs.apify.com/academy/web-scraping-for-beginners"}],"maxDepth":0,"maxPages":10}'

LangChain (Python).

from langchain_apify import ApifyActorsTool
tool = ApifyActorsTool("johto.data/url-to-markdown")

n8n, Make, Zapier. Use the Apify app or node, choose Run Actor, enter johto.data/url-to-markdown and paste the input JSON. Then map dataset fields into Sheets, a CRM or a database.

FAQ

How does the price compare? $0.80 per 1,000 pages, no start fee. Apify's own URL to Markdown lists $3 per 1,000 pages over plain HTTP on the free plan (checked Nov 8, 2026).

Does it render JavaScript? No. It uses plain HTTP, which is fast and cheap. Pages that need a browser to show content return less text.

Can it crawl a whole docs site? Yes. Set maxDepth above 0, cap it with maxPages and narrow it with includeUrlGlobs.

How long does a run take? The prefilled input finishes in about 3 seconds.

Responsible use

Convert only content you own, have permission to process, or may use under an applicable license. Public readability does not grant permission for commercial redistribution, model training or removal of attribution. Check each source's terms, robots.txt and content license. Preserve required attribution and license notices in downstream use.

The tool respects robots.txt by default. Do not disable that setting to bypass a source's restriction. Do not crawl behind logins or bypass access controls. Proxy configuration is for permitted access, not evading a source's prohibition.

Page text can contain personal data. You are responsible for a lawful basis and applicable privacy duties before collecting, storing or reusing it. Markdown conversion does not remove copyright, privacy or data-protection obligations.

Support

Open an issue in the Issues tab with the input URL and a description of the problem. Do not include API tokens or private data. Built and maintained by Johto Data.