Website to Markdown Crawler for AI, RAG & LLMs

Crawl any website, docs site or blog and get every page as clean Markdown: the article or documentation text only, without menus, sidebars, footers, cookie banners or scripts. Ready for ChatGPT, Claude, custom GPTs, RAG pipelines and vector databases.

• Main content only: Mozilla's Readability finds the real text; headings, lists, tables, code blocks and links stay intact
• Ready-made chunks for RAG: split at headings and paragraphs to the token size you choose, with overlap and the heading path of every chunk
• One combined Markdown file for the whole site and an llms.txt index, ready to upload as a knowledge file
• Finds pages through the sitemap and the links, stays inside the section you give it, skips duplicates
• Respects robots.txt, include and exclude patterns, depth and page limits
• No browser: fast and cheap, about 2 pages per second on default memory

What does this website to Markdown crawler do?

https://docs.python.org/3/tutorial/   → 17 pages (the whole tutorial)
https://react.dev/learn               → 51 pages
https://docs.apify.com/academy        → 200 pages
468 pages · 790,694 words · ~1.3 million tokens · 3,808 chunks of ~500 tokens

1. Give it start addresses: a whole site, a section (/docs/) or single pages.
2. It reads the sitemap and follows links inside that section, closest pages first, until your page limit.
3. Each page becomes Markdown with its title, description, word and token count, and chunks if you ask for them.

flowchart LR
    A[Start address] --> B[Sitemap + links<br/>inside the section]
    B --> C[Main content<br/>via Readability]
    C --> D[Markdown per page<br/>+ RAG chunks]
    D --> E[ALL_MARKDOWN file<br/>+ llms.txt]

Who is it for?

• AI developers feeding documentation into RAG, embeddings and vector databases (Pinecone, Qdrant, Weaviate, pgvector)
• Teams building custom GPTs and Claude projects that need a site's content as one knowledge file
• Support teams turning a help center into the source for an AI assistant
• Researchers and writers archiving blogs and essays as clean text
• Developers migrating a website's content into Markdown for a static site or a new CMS

What data do you get?

Pages (dataset, one row per page)

Field	What it tells you
url, title, description, h1, canonicalUrl, image	The page
markdown	The page's text as Markdown, starting with its heading
wordCount, tokens	Size; tokens are estimated at 4 characters each
chunks	When Chunk size is set: index, text, headings (where in the page it sits), tokens, url, title
language, author	From the page
extractedFrom	Main content, or the whole page when the page has no clear article
depth, foundVia, startUrl, crawledAt	How the crawler reached it

The key-value store also has ALL_MARKDOWN (every page in one Markdown file, each starting with its source address), LLMS_TXT (an llms.txt index of the pages), PAGES_CSV and RUN_SUMMARY (what was skipped and why).

A chunk from react.dev/learn/thinking-in-react:

{ "index": 2, "headings": ["Thinking in React", "Step 1: Break the UI into a component hierarchy"], "tokens": 239,
  "text": "displays and filters the list according to the user input.\n4.  `ProductCategoryRow` (green) displays a heading..." }

Example: 8 October 2026

4 start addresses, 468 pages in 3 minutes 38 seconds, with 500-token chunks.

Start address	Pages	How it found them
docs.python.org/3/tutorial/	17, the whole tutorial	links
react.dev/learn	51, the whole Learn section	links
docs.apify.com/academy	200 (the limit)	sitemap and links
paulgraham.com/articles.html	200 (the limit)	links

• 790,694 words, about 1.3 million tokens, 3,808 chunks.
• 461 pages had a clear main text; 7 index pages were kept whole.
• Skipped without charge: 35 links to files that are not pages (PDF, images), 1 duplicate page, and everything outside the four sections.

Pricing

$0.0005 per page saved as Markdown. Skipped pages, duplicates and pages that cannot be read are free.

• 1,000 pages: $0.50
• The run above: 468 pages, $0.23

Good to know

• JavaScript-only sites: pages that build all their text in the browser have no text in their HTML; they are listed in RUN_SUMMARY and not charged. Most docs sites, blogs and marketing sites serve their text in the HTML.
• Stay in a section: by default the crawler stays under the start address (/docs/ crawls /docs/...). Choose Whole website to follow every link on the site, or Only these pages to read just the listed addresses.
• Sites that block bots with a challenge page are listed in RUN_SUMMARY.
• robots.txt is respected by default; the start addresses you list are always read.
• Big crawls: thousands of pages work; for more speed give the run more memory. ALL_MARKDOWN holds up to 50 MB; the dataset always has every page.
• No login, no API key, no proxies needed.

Input

Field	Meaning
Websites or pages	Start addresses, one per line
Maximum pages per website	1 to 10,000 (default 50)
What to crawl	Section under the start address (default), whole website, or only these pages
Content	Main content (default) or whole page
Chunk size for RAG, Chunk overlap	Tokens; 0 = no chunks (default); overlap 50
Only pages matching, Skip pages matching	Patterns like /blog/**
Maximum link depth	Default 20
Use the sitemap, Respect robots.txt	Both on by default
Keep links, Keep images	Links on, images off
Skip duplicate pages	On by default
Pages at once	1 to 20 (default 5)

Integrations

Export to JSON, CSV or Excel, or connect through the Apify API, webhooks, Make, Zapier, n8n and LangChain. Upload ALL_MARKDOWN to a custom GPT or a Claude project, or send the chunks straight to your embedding model.