Website to Markdown Crawler for LLM & RAG
Pricing
from $1.00 / 1,000 page converteds
Website to Markdown Crawler for LLM & RAG
Crawl websites, sitemaps, or URL lists into clean Markdown for LLM, RAG, and AI pipelines. Readability main-content extraction; optional chunking for embeddings. Docs ingestion, help centers, blog-to-Markdown. Fast HTTP — $0.001/page; failed pages free.
Pricing
from $1.00 / 1,000 page converteds
Rating
0.0
(0)
Developer
Parth
Maintained by CommunityActor stats
0
Bookmarked
3
Total users
2
Monthly active users
8 days ago
Last modified
Categories
Features
Share
Turn any website into clean, LLM-ready Markdown. Give it a start URL, a list of URLs, or a sitemap — this Actor crawls the pages, strips menus, cookie banners, footers and ads with Mozilla Readability (the engine behind Firefox Reader View), and converts the main content to tidy Markdown with headings, lists, tables, code blocks and absolute links. Optional chunking splits each page into overlapping, heading-aware pieces ready for embeddings and vector databases.
Lightweight HTTP crawler (no headless browser) — fast and $0.001 per page converted ($1 per 1,000 pages). No subscription. Failed, blocked and skipped pages are free.
Who this is for
- LLM / RAG pipelines — ingest docs, help centers or knowledge bases into Pinecone, Qdrant, Weaviate, Chroma, OpenAI vector stores, LangChain or LlamaIndex
- Docs ingestion — crawl
/docs(or a sitemap) into structured Markdown + metadata for search and chatbots - Blog → Markdown — export a blog or changelog as clean Markdown for migration, archiving or fine-tuning datasets
- AI agents & custom GPTs — keep an agent's knowledge of a product site up to date
- Content / SEO research — word counts, metadata and clean text without the chrome
Features
- Three modes: single pages, crawl (follow links with depth and scope control) or sitemap (every URL in
sitemap.xml) - Main-content extraction (Readability) or full page minus navigation / header / footer
- GitHub-flavored Markdown: tables, fenced code, task lists; absolute links; optional images
- Chunking for RAG with overlap, split on headings and paragraphs
- Metadata: title, description, author, published date, language, site name, OG image, canonical URL
estimatedTokensper page to plan LLM costs- Include / exclude URL patterns, query-string de-duplication, robots.txt support
- Remove extra elements with your own CSS selectors
Example inputs
1) RAG / docs ingestion (crawl a docs path)
{"startUrls": [{ "url": "https://crawlee.dev/js/docs/quick-start" }],"crawlMode": "crawl","crawlScope": "path","maxPages": 200,"maxDepth": 3,"contentMode": "main","chunkSize": 2000,"chunkOverlap": 200}
2) Blog → Markdown (single section)
{"startUrls": [{ "url": "https://blog.example.com/" }],"crawlMode": "crawl","crawlScope": "host","includeUrlPatterns": ["https://blog.example.com/**"],"excludeUrlPatterns": ["https://blog.example.com/tag/**", "https://blog.example.com/author/**"],"maxPages": 100,"maxDepth": 2,"contentMode": "main","minWordCount": 100,"chunkSize": 0}
3) Sitemap → full site Markdown
{"startUrls": [{ "url": "https://docs.example.com/" }],"crawlMode": "sitemap","maxPages": 500,"contentMode": "main","chunkSize": 1500,"chunkOverlap": 150,"respectRobotsTxt": true}
4) Single pages only (URL list)
{"startUrls": [{ "url": "https://example.com/pricing" },{ "url": "https://example.com/faq" }],"crawlMode": "single","contentMode": "main","includeText": true}
Output example
{"url": "https://crawlee.dev/js/docs/quick-start","statusCode": 200,"title": "Quick Start | Crawlee for JavaScript","description": "Crawlee helps you build and maintain your crawlers…","language": "en","wordCount": 755,"estimatedTokens": 1290,"markdown": "# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two…\n\n## Choose your crawler\n\n…","chunks": ["# Quick Start\n\nWith this short tutorial…", "## Installation\n\n…"],"crawledAt": "2026-10-23T06:40:00.000Z"}
Download results as JSON, JSONL, CSV or Excel, or read them via the Apify API / Python & JS clients, LangChain's ApifyDatasetLoader, Make, Zapier or n8n.
How to run
- Open this Actor in Apify Console (or call it via API / client).
- Paste your start URL(s), pick Crawl, Sitemap or Single pages.
- Optional: set
chunkSize(e.g. 1500–2000) for RAG-ready chunks. - Set Max pages (and optionally Max cost per run) so spend stays predictable.
- Run — each successfully converted page is one
page-convertedevent.
Via API (Python):
from apify_client import ApifyClientclient = ApifyClient(token="YOUR_TOKEN")run = client.actor("parth-tools/website-to-markdown-crawler").call(run_input={"startUrls": [{"url": "https://crawlee.dev/js/docs/quick-start"}],"crawlMode": "crawl","crawlScope": "path","maxPages": 50,"chunkSize": 2000,"chunkOverlap": 200,})for item in client.dataset(run["defaultDatasetId"]).iterate_items():print(item["url"], item.get("wordCount"), len(item.get("markdown") or ""))
Pricing
Pay-per-event, no monthly fee:
| Event | Price | When |
|---|---|---|
| Page converted | $0.001 ($1 per 1,000 pages) | Each page saved with its Markdown |
| Actor start | $0.00005 | Once per run (scales with memory GB) |
Pages that fail to load, return errors, aren't HTML, are blocked by robots.txt or are under your minimum word count are free. Use Apify's Max cost per run to cap spending — the crawler stops cleanly at the limit.
FAQ
Does it run JavaScript? No — it reads the server-rendered HTML, which covers most documentation sites, blogs, news, help centers and marketing sites, and keeps it fast and cheap. Pages that render everything client-side may come back thin.
How do I crawl only the docs section? Set Crawl scope to Same path prefix and start from e.g. https://example.com/docs/, or use Only URLs matching with https://example.com/docs/**.
How big should chunks be? 1,000–2,000 characters (~250–500 tokens) with 100–200 overlap is a good default for most embedding models.
Can I get plain text or HTML too? Yes, enable Also output plain text and/or Also output cleaned HTML.
Is it allowed to crawl any site? You are responsible for crawling only content you're permitted to use. The Actor respects robots.txt by default.