Website to Markdown for LLM & RAG: Fast Content Crawler
Pricing
from $1.00 / 1,000 page extracteds
Website to Markdown for LLM & RAG: Fast Content Crawler
Website to clean Markdown for ChatGPT, RAG & vector DBs. Up to 10x faster than browser crawlers (benchmarked), flat $1 per 1,000 pages, free chunking. Sitemaps, robots.txt.
Pricing
from $1.00 / 1,000 page extracteds
Rating
0.0
(0)
Developer
Datacraft Tools
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
2 days ago
Last modified
Categories
Features
Share
Turn any public website into clean, LLM-ready Markdown: the main content only, without menus, cookie banners, footers or ads.
What makes it different
| This actor | Typical browser-based crawler | |
|---|---|---|
| Engine | Lightweight HTTP, no headless Chrome | Full headless browser for every page |
| Speed (our benchmark) | Up to 10x faster (see below) | Loads JS, CSS, images and fonts for every page |
| Price | Flat $1 per 1,000 pages + $0.01 per run | Billed by compute units, hard to predict |
| Chunking for RAG | Built in; you pay per page, not per chunk | Usually a separate step |
| Example: 1,000-page docs site | ≈ $1.01 | Depends on memory and runtime |
In short: if a site serves its content as HTML (docs, blogs, help centers, news and marketing sites), you get the same clean Markdown much faster and for a predictable price.
Benchmark
We crawled the same 30 pages with this actor and with a headless Chromium browser (Playwright), both with 5 parallel requests:
| Site | This actor | Headless browser | Speed-up |
|---|---|---|---|
| MDN Web Docs (30 pages) | 1.0 s | 9.5 s | 9.8x |
| MDN Web Docs, cold start | 3.2 s | 8.1 s | 2.5x |
| Simple static blog (30 pages) | 1.9 s | 3.4 s | 1.8x |
| Heavy news site | finished | browser timed out | – |
The heavier the site (scripts, ads, trackers), the bigger the gap. The actor also needs less memory than a browser, which keeps the price low.
Features
- Up to 10x faster than headless-browser crawling. It uses a pure HTTP crawler with no Chrome, and a flat per-page price means no compute-unit surprises.
- Works with ChatGPT and custom GPTs, LangChain, LlamaIndex and vector databases (Pinecone, Qdrant, Weaviate, Chroma).
- Main content extraction. Boilerplate (navigation, sidebars, footers, comments) is removed; headings, lists, code blocks and tables are kept as Markdown.
- Built-in chunking. Set
chunkSizeto get ready-to-embed chunks with overlap. You pay per page, not per chunk. - Smart crawling. Stays on the start domains, follows links up to
maxCrawlDepth, can discover pages from sitemap.xml, and supports include/exclude URL patterns. - Polite by default. Respects robots.txt, deduplicates URLs, skips binary files and lets you set concurrency.
- Rich metadata. Title, description, author, publish date, site name, language, word count and crawl depth.
Use cases
- Build a RAG chatbot over your docs, help center or blog
- Feed company or competitor websites to an LLM for research
- Create training or eval datasets from documentation
- Monitor website content changes (schedule the actor)
- Move a website's content into Notion or Obsidian as Markdown
Input example
{"startUrls": [{ "url": "https://docs.example.com/" }],"useSitemap": true,"maxPages": 500,"includeUrlGlobs": ["https://docs.example.com/*"],"chunkSize": 2000,"chunkOverlap": 200}
Output example
{"url": "https://docs.example.com/getting-started","title": "Getting started","description": "Install and configure…","language": "en","markdown": "# Getting started\n\nInstall the package…","wordCount": 812,"chunkIndex": 0,"chunkCount": 3,"depth": 1}
Limits
This crawler doesn't run JavaScript. Pages that render their content only in the browser (some single-page apps) may come back empty. Most docs sites, blogs, news sites, marketing sites and knowledge bases work well. Pages behind a login are not supported.
Pricing
A small fee per run start plus a fee per page extracted, no matter how many chunks a page produces. There are no monthly fees.