Website to Markdown Crawler for LLM & RAG avatar

Website to Markdown Crawler for LLM & RAG

Pricing

from $1.00 / 1,000 page converteds

Go to Apify Store
Website to Markdown Crawler for LLM & RAG

Website to Markdown Crawler for LLM & RAG

Crawl websites, sitemaps, or URL lists into clean Markdown for LLM, RAG, and AI pipelines. Readability main-content extraction; optional chunking for embeddings. Docs ingestion, help centers, blog-to-Markdown. Fast HTTP — $0.001/page; failed pages free.

Pricing

from $1.00 / 1,000 page converteds

Rating

0.0

(0)

Developer

Parth

Parth

Maintained by Community

Actor stats

0

Bookmarked

3

Total users

2

Monthly active users

8 days ago

Last modified

Features

Limited permissions

Share

Turn any website into clean, LLM-ready Markdown. Give it a start URL, a list of URLs, or a sitemap — this Actor crawls the pages, strips menus, cookie banners, footers and ads with Mozilla Readability (the engine behind Firefox Reader View), and converts the main content to tidy Markdown with headings, lists, tables, code blocks and absolute links. Optional chunking splits each page into overlapping, heading-aware pieces ready for embeddings and vector databases.

Lightweight HTTP crawler (no headless browser) — fast and $0.001 per page converted ($1 per 1,000 pages). No subscription. Failed, blocked and skipped pages are free.

Who this is for

  • LLM / RAG pipelines — ingest docs, help centers or knowledge bases into Pinecone, Qdrant, Weaviate, Chroma, OpenAI vector stores, LangChain or LlamaIndex
  • Docs ingestion — crawl /docs (or a sitemap) into structured Markdown + metadata for search and chatbots
  • Blog → Markdown — export a blog or changelog as clean Markdown for migration, archiving or fine-tuning datasets
  • AI agents & custom GPTs — keep an agent's knowledge of a product site up to date
  • Content / SEO research — word counts, metadata and clean text without the chrome

Features

  • Three modes: single pages, crawl (follow links with depth and scope control) or sitemap (every URL in sitemap.xml)
  • Main-content extraction (Readability) or full page minus navigation / header / footer
  • GitHub-flavored Markdown: tables, fenced code, task lists; absolute links; optional images
  • Chunking for RAG with overlap, split on headings and paragraphs
  • Metadata: title, description, author, published date, language, site name, OG image, canonical URL
  • estimatedTokens per page to plan LLM costs
  • Include / exclude URL patterns, query-string de-duplication, robots.txt support
  • Remove extra elements with your own CSS selectors

Example inputs

1) RAG / docs ingestion (crawl a docs path)

{
"startUrls": [{ "url": "https://crawlee.dev/js/docs/quick-start" }],
"crawlMode": "crawl",
"crawlScope": "path",
"maxPages": 200,
"maxDepth": 3,
"contentMode": "main",
"chunkSize": 2000,
"chunkOverlap": 200
}

2) Blog → Markdown (single section)

{
"startUrls": [{ "url": "https://blog.example.com/" }],
"crawlMode": "crawl",
"crawlScope": "host",
"includeUrlPatterns": ["https://blog.example.com/**"],
"excludeUrlPatterns": ["https://blog.example.com/tag/**", "https://blog.example.com/author/**"],
"maxPages": 100,
"maxDepth": 2,
"contentMode": "main",
"minWordCount": 100,
"chunkSize": 0
}

3) Sitemap → full site Markdown

{
"startUrls": [{ "url": "https://docs.example.com/" }],
"crawlMode": "sitemap",
"maxPages": 500,
"contentMode": "main",
"chunkSize": 1500,
"chunkOverlap": 150,
"respectRobotsTxt": true
}

4) Single pages only (URL list)

{
"startUrls": [
{ "url": "https://example.com/pricing" },
{ "url": "https://example.com/faq" }
],
"crawlMode": "single",
"contentMode": "main",
"includeText": true
}

Output example

{
"url": "https://crawlee.dev/js/docs/quick-start",
"statusCode": 200,
"title": "Quick Start | Crawlee for JavaScript",
"description": "Crawlee helps you build and maintain your crawlers…",
"language": "en",
"wordCount": 755,
"estimatedTokens": 1290,
"markdown": "# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two…\n\n## Choose your crawler\n\n…",
"chunks": ["# Quick Start\n\nWith this short tutorial…", "## Installation\n\n…"],
"crawledAt": "2026-10-23T06:40:00.000Z"
}

Download results as JSON, JSONL, CSV or Excel, or read them via the Apify API / Python & JS clients, LangChain's ApifyDatasetLoader, Make, Zapier or n8n.

How to run

  1. Open this Actor in Apify Console (or call it via API / client).
  2. Paste your start URL(s), pick Crawl, Sitemap or Single pages.
  3. Optional: set chunkSize (e.g. 1500–2000) for RAG-ready chunks.
  4. Set Max pages (and optionally Max cost per run) so spend stays predictable.
  5. Run — each successfully converted page is one page-converted event.

Via API (Python):

from apify_client import ApifyClient
client = ApifyClient(token="YOUR_TOKEN")
run = client.actor("parth-tools/website-to-markdown-crawler").call(run_input={
"startUrls": [{"url": "https://crawlee.dev/js/docs/quick-start"}],
"crawlMode": "crawl",
"crawlScope": "path",
"maxPages": 50,
"chunkSize": 2000,
"chunkOverlap": 200,
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
print(item["url"], item.get("wordCount"), len(item.get("markdown") or ""))

Pricing

Pay-per-event, no monthly fee:

EventPriceWhen
Page converted$0.001 ($1 per 1,000 pages)Each page saved with its Markdown
Actor start$0.00005Once per run (scales with memory GB)

Pages that fail to load, return errors, aren't HTML, are blocked by robots.txt or are under your minimum word count are free. Use Apify's Max cost per run to cap spending — the crawler stops cleanly at the limit.

FAQ

Does it run JavaScript? No — it reads the server-rendered HTML, which covers most documentation sites, blogs, news, help centers and marketing sites, and keeps it fast and cheap. Pages that render everything client-side may come back thin.

How do I crawl only the docs section? Set Crawl scope to Same path prefix and start from e.g. https://example.com/docs/, or use Only URLs matching with https://example.com/docs/**.

How big should chunks be? 1,000–2,000 characters (~250–500 tokens) with 100–200 overlap is a good default for most embedding models.

Can I get plain text or HTML too? Yes, enable Also output plain text and/or Also output cleaned HTML.

Is it allowed to crawl any site? You are responsible for crawling only content you're permitted to use. The Actor respects robots.txt by default.