Website to Markdown for LLM & RAG: Fast Content Crawler avatar

Website to Markdown for LLM & RAG: Fast Content Crawler

Pricing

from $1.00 / 1,000 page extracteds

Go to Apify Store
Website to Markdown for LLM & RAG: Fast Content Crawler

Website to Markdown for LLM & RAG: Fast Content Crawler

Website to clean Markdown for ChatGPT, RAG & vector DBs. Up to 10x faster than browser crawlers (benchmarked), flat $1 per 1,000 pages, free chunking. Sitemaps, robots.txt.

Pricing

from $1.00 / 1,000 page extracteds

Rating

0.0

(0)

Developer

Datacraft Tools

Datacraft Tools

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

2 days ago

Last modified

Features

Limited permissions

Share

Turn any public website into clean, LLM-ready Markdown: the main content only, without menus, cookie banners, footers or ads.

What makes it different

This actorTypical browser-based crawler
EngineLightweight HTTP, no headless ChromeFull headless browser for every page
Speed (our benchmark)Up to 10x faster (see below)Loads JS, CSS, images and fonts for every page
PriceFlat $1 per 1,000 pages + $0.01 per runBilled by compute units, hard to predict
Chunking for RAGBuilt in; you pay per page, not per chunkUsually a separate step
Example: 1,000-page docs site≈ $1.01Depends on memory and runtime

In short: if a site serves its content as HTML (docs, blogs, help centers, news and marketing sites), you get the same clean Markdown much faster and for a predictable price.

Benchmark

We crawled the same 30 pages with this actor and with a headless Chromium browser (Playwright), both with 5 parallel requests:

SiteThis actorHeadless browserSpeed-up
MDN Web Docs (30 pages)1.0 s9.5 s9.8x
MDN Web Docs, cold start3.2 s8.1 s2.5x
Simple static blog (30 pages)1.9 s3.4 s1.8x
Heavy news sitefinishedbrowser timed out–

The heavier the site (scripts, ads, trackers), the bigger the gap. The actor also needs less memory than a browser, which keeps the price low.

Features

  • Up to 10x faster than headless-browser crawling. It uses a pure HTTP crawler with no Chrome, and a flat per-page price means no compute-unit surprises.
  • Works with ChatGPT and custom GPTs, LangChain, LlamaIndex and vector databases (Pinecone, Qdrant, Weaviate, Chroma).
  • Main content extraction. Boilerplate (navigation, sidebars, footers, comments) is removed; headings, lists, code blocks and tables are kept as Markdown.
  • Built-in chunking. Set chunkSize to get ready-to-embed chunks with overlap. You pay per page, not per chunk.
  • Smart crawling. Stays on the start domains, follows links up to maxCrawlDepth, can discover pages from sitemap.xml, and supports include/exclude URL patterns.
  • Polite by default. Respects robots.txt, deduplicates URLs, skips binary files and lets you set concurrency.
  • Rich metadata. Title, description, author, publish date, site name, language, word count and crawl depth.

Use cases

  • Build a RAG chatbot over your docs, help center or blog
  • Feed company or competitor websites to an LLM for research
  • Create training or eval datasets from documentation
  • Monitor website content changes (schedule the actor)
  • Move a website's content into Notion or Obsidian as Markdown

Input example

{
"startUrls": [{ "url": "https://docs.example.com/" }],
"useSitemap": true,
"maxPages": 500,
"includeUrlGlobs": ["https://docs.example.com/*"],
"chunkSize": 2000,
"chunkOverlap": 200
}

Output example

{
"url": "https://docs.example.com/getting-started",
"title": "Getting started",
"description": "Install and configure…",
"language": "en",
"markdown": "# Getting started\n\nInstall the package…",
"wordCount": 812,
"chunkIndex": 0,
"chunkCount": 3,
"depth": 1
}

Limits

This crawler doesn't run JavaScript. Pages that render their content only in the browser (some single-page apps) may come back empty. Most docs sites, blogs, news sites, marketing sites and knowledge bases work well. Pages behind a login are not supported.

Pricing

A small fee per run start plus a fee per page extracted, no matter how many chunks a page produces. There are no monthly fees.