Website to Markdown Crawler for LLM & RAG

Turn any website into clean, LLM-ready Markdown. Give it a URL, a list of URLs or a sitemap — the Actor crawls the pages, strips menus, cookie banners, footers and ads with Mozilla Readability (the engine behind Firefox Reader View), and converts the main content to tidy Markdown with headings, lists, tables, code blocks and absolute links. Optional chunking splits each page into overlapping, heading-aware pieces ready for embeddings and vector databases.

It's a lightweight HTTP crawler (no headless browser), so it's fast and costs just $1 per 1,000 pages — no subscription, failed pages are free.

Use cases

• 🤖 RAG & chatbots — feed documentation, help centers or blogs into Pinecone, Qdrant, Weaviate, Chroma, OpenAI vector stores, LangChain or LlamaIndex
• 🧠 AI agents & custom GPTs — give an agent fresh knowledge of a website
• 📚 Docs migration & archiving — export a site's content to Markdown files
• 🔍 Content analysis — word counts, metadata and clean text for SEO or research

Features

• Three modes: single pages, crawl (follow links with depth and scope control) or sitemap (every URL in sitemap.xml)
• Main-content extraction (Readability) or full page minus navigation/header/footer
• GitHub-flavored Markdown: tables, fenced code, task lists; absolute links; optional images
• Chunking for RAG with overlap, split on headings and paragraphs
• Metadata: title, description, author, published date, language, site name, OG image, canonical URL
• estimatedTokens per page to plan LLM costs
• Include/exclude URL patterns, query-string de-duplication, robots.txt support
• Remove extra elements with your own CSS selectors

Input example

{
    "startUrls": [{ "url": "https://crawlee.dev/js/docs/quick-start" }],
    "crawlMode": "crawl",
    "crawlScope": "path",
    "maxPages": 200,
    "maxDepth": 3,
    "contentMode": "main",
    "chunkSize": 2000,
    "chunkOverlap": 200
}

Output example

{
    "url": "https://crawlee.dev/js/docs/quick-start",
    "statusCode": 200,
    "title": "Quick Start | Crawlee for JavaScript",
    "description": "Crawlee helps you build and maintain your crawlers…",
    "language": "en",
    "wordCount": 755,
    "estimatedTokens": 1290,
    "markdown": "# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two…\n\n## Choose your crawler\n\n…",
    "chunks": ["# Quick Start\n\nWith this short tutorial…", "## Installation\n\n…"],
    "crawledAt": "2026-10-23T06:40:00.000Z"
}

Download results as JSON, JSONL, CSV or Excel, or read them via the Apify API / Python & JS clients, LangChain's ApifyDatasetLoader, Make, Zapier or n8n.

Pricing

Pay-per-event, no monthly fee:

Event	Price	When
Page converted	$0.001 ($1 per 1,000 pages)	Each page saved with its Markdown
Actor start	$0.00005	Once per run

Pages that fail to load, return errors, aren't HTML, are blocked by robots.txt or are under your minimum word count are free. Set "Max cost per run" to cap spending — the crawler stops cleanly at the limit.

FAQ

Does it run JavaScript? No — it reads the server-rendered HTML, which covers most documentation sites, blogs, news, help centers and marketing sites, and keeps it fast and cheap. Pages that render everything client-side may come back thin.

How do I crawl only the docs section? Set Crawl scope to Same path prefix and start from e.g. https://example.com/docs/, or use Only URLs matching with https://example.com/docs/**.

How big should chunks be? 1,000–2,000 characters (~250–500 tokens) with 100–200 overlap is a good default for most embedding models.

Can I get plain text or HTML too? Yes, enable Also output plain text and/or Also output cleaned HTML.

Is it allowed to crawl any site? You are responsible for crawling only content you're permitted to use. The Actor respects robots.txt by default.