Website to Markdown Crawler for LLM & RAG
Pricing
from $1.00 / 1,000 page converted to markdowns
Website to Markdown Crawler for LLM & RAG
Turn a website into clean, LLM-ready Markdown for RAG pipelines, AI agents and vector databases. Keeps main content, tables and code; drops menus and footers. Adds metadata, token estimates and optional chunks. Fast HTTP, robots.txt respected. Failed or empty pages are free.
Pricing
from $1.00 / 1,000 page converted to markdowns
Rating
0.0
(0)
Developer
LiveFree Analytics
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
18 days ago
Last modified
Categories
Features
Share
Website to Markdown for LLMs & RAG: Clean Docs Crawler
Point it at a docs site, blog or help center. Get clean Markdown for every page, with menus, footers and ads removed, and pre-cut into chunks ready for ChatGPT, Claude or your vector database. Each page shows its token count, so you know the cost before you embed.
Who it's for
- AI builders: load documentation into a RAG pipeline (LangChain, LlamaIndex, Pinecone, OpenAI Assistants, custom GPTs)
- Support teams: turn your help center into a knowledge base for an AI support bot
- AI agents: give a model a clean, readable copy of any web page through the Apify MCP server
- Content teams: archive a site or spot changed pages (compare
contentHashbetween runs)
What you get (real output row, text shortened)
{"url": "https://docs.python.org/3/tutorial/appetite.html","statusCode": 200,"depth": 1,"title": "1. Whetting Your Appetite","language": "en","extractionMode": "article","markdown": "# 1. Whetting Your Appetite\n\nIf you do much work on computers, eventually you find that there's some task you'd like to automate...","wordCount": 730,"tokensEstimate": 1146,"contentHash": "4f1c0e...","chunks": [{ "index": 0, "text": "# 1. Whetting Your Appetite\n\n..." }]}
Clean GitHub-style Markdown with tables, code blocks and absolute links. Export to JSON, CSV or Excel, or pull it by API.
Try it in 1 minute
-
Put a docs or blog URL in Start URLs.
-
Set Max pages and, for RAG, a Chunk size (2,000 characters is a good start).
-
Click Start.
{ "startUrls": ["https://docs.python.org/3/tutorial/"], "maxCrawlDepth": 2, "maxPages": 200, "chunkSize": 2000 }
Popular setups
| Goal | Settings |
|---|---|
| One section of a docs site | crawlScope: samePath (default) |
| A whole site, fast | useSitemaps: true, crawlScope: sameHostname |
| Fewest tokens | keepImages: false, keepLinks: false |
| Pages for a vector DB | chunkSize: 2000, chunkOverlap: 200 |
| A single page for an agent | maxPages: 1, maxCrawlDepth: 0 |
| Whole page, not just the article | contentMode: fullPage |
Price
$2.00 per 1,000 pages converted on Apify's Free plan ($1.70 Starter, $1.40 Scale, $1.00 Business). Compute and chunking are included.
- No start fee from us. Only Apify's standard minimum start event (a fraction of a cent).
- Pages that fail, are blocked, or are too short (under
minContentChars) are free. - If no page could be converted, the run fails with a clear message and costs nothing. That usually means the site needs JavaScript.
- The run stops cleanly at your Max total charge and never stores results you didn't pay for.
All settings
| Field | Default | What it does |
|---|---|---|
startUrls | – (required) | Where to start |
maxCrawlDepth / maxPages | 1 / 100 | How far to crawl |
crawlScope | samePath | samePath, sameHostname or sameDomain |
useSitemaps | false | Also find pages through sitemaps |
includeUrlPatterns / excludeUrlPatterns | – | Regex filters |
contentMode | article | Main content only, or fullPage |
removeSelectors | – | Extra CSS selectors to drop |
keepImages / keepLinks | true / true | Drop them to save tokens |
chunkSize / chunkOverlap | 0 (off) / 200 | RAG chunks in characters |
includeText / includeHtml / includeLinksList | false | Extra output fields |
minContentChars | 100 | Skip near-empty pages (free) |
respectRobotsTxt / maxConcurrency / requestDelayMs | true / 10 / 200 | Politeness and speed |
requestTimeoutSecs / maxRetries / userAgent / proxyConfiguration | 30 / 2 / – / off | Advanced |
Use it from code, no-code tools or AI agents
- API:
POST https://api.apify.com/v2/acts/blond_terrarium~website-to-markdown-llm/run-sync-get-dataset-items?token=YOUR_TOKEN - LangChain / LlamaIndex: use the Apify dataset loader on the run's dataset
- Make, Zapier, n8n through Apify integrations
- AI agents (MCP):
https://mcp.apify.com?tools=blond_terrarium/website-to-markdown-llm. Ask "Read docs.example.com/start and summarize it".
FAQ
Does it run JavaScript? No. That's why it's fast and cheap. For apps that only show content after JS runs, use a browser-based crawler. Does it respect robots.txt? Yes, by default, including Crawl-delay. Pages behind a login? No, public pages only. Support? Use the Issues tab. We read every one.
Related tools
PDF to Text & Markdown (PDFs for the same pipeline) · Sitemap URL Extractor · Google News Scraper
Built and maintained by LiveFree Analytics.