Website to Markdown Crawler for LLM & RAG avatar

Website to Markdown Crawler for LLM & RAG

Pricing

from $1.00 / 1,000 page converted to markdowns

Go to Apify Store
Website to Markdown Crawler for LLM & RAG

Website to Markdown Crawler for LLM & RAG

Turn a website into clean, LLM-ready Markdown for RAG pipelines, AI agents and vector databases. Keeps main content, tables and code; drops menus and footers. Adds metadata, token estimates and optional chunks. Fast HTTP, robots.txt respected. Failed or empty pages are free.

Pricing

from $1.00 / 1,000 page converted to markdowns

Rating

0.0

(0)

Developer

LiveFree Analytics

LiveFree Analytics

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

18 days ago

Last modified

Features

Limited permissions

Share

Website to Markdown for LLMs & RAG: Clean Docs Crawler

Point it at a docs site, blog or help center. Get clean Markdown for every page, with menus, footers and ads removed, and pre-cut into chunks ready for ChatGPT, Claude or your vector database. Each page shows its token count, so you know the cost before you embed.

Who it's for

  • AI builders: load documentation into a RAG pipeline (LangChain, LlamaIndex, Pinecone, OpenAI Assistants, custom GPTs)
  • Support teams: turn your help center into a knowledge base for an AI support bot
  • AI agents: give a model a clean, readable copy of any web page through the Apify MCP server
  • Content teams: archive a site or spot changed pages (compare contentHash between runs)

What you get (real output row, text shortened)

{
"url": "https://docs.python.org/3/tutorial/appetite.html",
"statusCode": 200,
"depth": 1,
"title": "1. Whetting Your Appetite",
"language": "en",
"extractionMode": "article",
"markdown": "# 1. Whetting Your Appetite\n\nIf you do much work on computers, eventually you find that there's some task you'd like to automate...",
"wordCount": 730,
"tokensEstimate": 1146,
"contentHash": "4f1c0e...",
"chunks": [{ "index": 0, "text": "# 1. Whetting Your Appetite\n\n..." }]
}

Clean GitHub-style Markdown with tables, code blocks and absolute links. Export to JSON, CSV or Excel, or pull it by API.

Try it in 1 minute

  1. Put a docs or blog URL in Start URLs.

  2. Set Max pages and, for RAG, a Chunk size (2,000 characters is a good start).

  3. Click Start.

    { "startUrls": ["https://docs.python.org/3/tutorial/"], "maxCrawlDepth": 2, "maxPages": 200, "chunkSize": 2000 }

GoalSettings
One section of a docs sitecrawlScope: samePath (default)
A whole site, fastuseSitemaps: true, crawlScope: sameHostname
Fewest tokenskeepImages: false, keepLinks: false
Pages for a vector DBchunkSize: 2000, chunkOverlap: 200
A single page for an agentmaxPages: 1, maxCrawlDepth: 0
Whole page, not just the articlecontentMode: fullPage

Price

$2.00 per 1,000 pages converted on Apify's Free plan ($1.70 Starter, $1.40 Scale, $1.00 Business). Compute and chunking are included.

  • No start fee from us. Only Apify's standard minimum start event (a fraction of a cent).
  • Pages that fail, are blocked, or are too short (under minContentChars) are free.
  • If no page could be converted, the run fails with a clear message and costs nothing. That usually means the site needs JavaScript.
  • The run stops cleanly at your Max total charge and never stores results you didn't pay for.

All settings

FieldDefaultWhat it does
startUrls– (required)Where to start
maxCrawlDepth / maxPages1 / 100How far to crawl
crawlScopesamePathsamePath, sameHostname or sameDomain
useSitemapsfalseAlso find pages through sitemaps
includeUrlPatterns / excludeUrlPatterns–Regex filters
contentModearticleMain content only, or fullPage
removeSelectors–Extra CSS selectors to drop
keepImages / keepLinkstrue / trueDrop them to save tokens
chunkSize / chunkOverlap0 (off) / 200RAG chunks in characters
includeText / includeHtml / includeLinksListfalseExtra output fields
minContentChars100Skip near-empty pages (free)
respectRobotsTxt / maxConcurrency / requestDelayMstrue / 10 / 200Politeness and speed
requestTimeoutSecs / maxRetries / userAgent / proxyConfiguration30 / 2 / – / offAdvanced

Use it from code, no-code tools or AI agents

  • API: POST https://api.apify.com/v2/acts/blond_terrarium~website-to-markdown-llm/run-sync-get-dataset-items?token=YOUR_TOKEN
  • LangChain / LlamaIndex: use the Apify dataset loader on the run's dataset
  • Make, Zapier, n8n through Apify integrations
  • AI agents (MCP): https://mcp.apify.com?tools=blond_terrarium/website-to-markdown-llm. Ask "Read docs.example.com/start and summarize it".

FAQ

Does it run JavaScript? No. That's why it's fast and cheap. For apps that only show content after JS runs, use a browser-based crawler. Does it respect robots.txt? Yes, by default, including Crawl-delay. Pages behind a login? No, public pages only. Support? Use the Issues tab. We read every one.

PDF to Text & Markdown (PDFs for the same pipeline) · Sitemap URL Extractor · Google News Scraper

Built and maintained by LiveFree Analytics.