Website to Markdown Crawler for LLMs and RAG

Turn a documentation site, help center or blog into clean Markdown for your LLM, RAG pipeline or AI agent in one run.

• Main content only. Menus, footers and cookie notes are removed. Headings, links, tables and code blocks stay.
• Ready for RAG. Every page comes with chunks, a heading path and a token estimate, plus llms.txt for the whole site.
• $1.50 per 1,000 pages on the free plan, less on paid plans. Plain HTTP with no browser. The default 30-page run took 23 seconds on the platform.

One output row, shortened:

{
  "url": "https://crawlee.dev/js/docs/quick-start",
  "title": "Quick Start",
  "language": "en",
  "markdown": "# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two ...",
  "wordCount": 575,
  "tokenEstimate": 4346,
  "chunkCount": 20,
  "chunks": [{ "index": 0, "headingPath": ["Quick Start"], "text": "# Quick Start\n\nWith this short tutorial ...", "tokenEstimate": 73 }]
}

The full row is in Example output.

How the crawler works

This Actor crawls a website and turns each page into clean Markdown for LLMs, RAG pipelines, AI agents and AI training. It uses plain HTTP requests and no browser, so it is fast and cheap. It reads the site's XML sitemaps first, then follows links inside the scope you set. It keeps only the main content of each page, with headings, links, lists, tables and code blocks.

Each page also gets its title, description, language, word count and a token estimate. Optional RAG chunks come with their heading path and token estimate. One run also saves llms.txt and llms-full.txt files for the site.

Use it on your own website, your documentation, or sites whose content you may reuse.

What you get

Field	What it holds
markdown	Main content as Markdown: headings, links, lists, tables (GitHub style) and fenced code blocks with their language
title, description, language	Page title, meta description and the language from the HTML lang attribute
wordCount, tokenEstimate	Words in the page and estimated LLM tokens
chunks, chunkCount	Chunks for a vector database, each with headingPath, text, tokenEstimate and charCount
contentHash, changeStatus	A hash of the Markdown, and "new" or "changed" in change-only mode
url, loadedUrl, httpStatus	Requested URL, URL after redirects, HTTP status
contentSelectorMatched	Whether your content selector matched on the page. Null when you set none
sourceType	html for web pages, pdf for linked PDF files
contentWarning	needs-javascript or little-text when a page gave under 50 words and no code block. Null for normal pages
canonicalUrl	The page's canonical URL, for removing duplicates. Null when the page sets none
metadata	Site name, keywords, publish and change dates, Open Graph type and image, and the Last-Modified, ETag and Content-Type headers. No author names
depth, discoveredBy, sitemapLastmod	Link depth, how the page was found (start, sitemap, link or llms.txt) and the sitemap's last-modified date

The run's key-value store holds two more files:

• llms.txt: an index of all converted pages in the llmstxt.org format, with title, link and description.
• llms-full.txt: the Markdown of all pages in one file, each with its source URL. It stops at 20 MB.

Why use this crawler

• Price. $1.50 per 1,000 pages on the free plan, plus a small start fee per run. You pay per page saved, so the cost is known before you start.
• Speed. HTTP only, with no browser to start. A 100-page run took 96 seconds on the platform at the default polite rate.
• Sitemap first. The crawler reads the XML sitemaps before it follows links. It finds pages that no link points to, and it finds them sooner.
• RAG chunks in the same row. No second tool is needed to split pages. A new chunk starts at each H1 to H3 heading and when the size limit is reached. Code blocks and tables stay whole when they fit.
• llms.txt included. The same run builds llms.txt and llms-full.txt for the site.
• Change-only mode. Schedule the Actor to keep a knowledge base fresh. It saves only pages that are new or whose content changed.

This crawler does not run JavaScript. Pages that build their content in the browser may come back empty or short. For those sites, use a browser-based crawler.

How to use it

1. Add one or more start URLs.
2. Choose the crawl scope. By default the crawl stays inside the folder of each start URL. For example, https://example.com/docs/intro crawls pages under https://example.com/docs/.
3. Set Maximum pages. The default is 30, so a first run stays small. Set 0 for no limit.
4. Run the Actor. Download the pages as JSON, CSV or Excel, or read llms.txt and llms-full.txt from the key-value store.

Input

Field	Default	What it does
Start URLs	crawlee.dev quick start page	Pages where the crawl starts
Maximum pages	30	Stop after this many pages. Raise it for a full site. 0 means no limit
Crawl scope	Same folder	Same folder, same domain, or domain and subdomains
Include and exclude URL patterns	none	Globs. * matches any text except /, ** matches any text. Include patterns replace the scope
Read sitemaps first	on	Queue in-scope sitemap pages before following links
Use the site's llms.txt	off	Also crawl the in-scope pages listed in the site's own /llms.txt
Follow links	on	Follow in-scope links on each page
Maximum link depth	20	Start URLs are depth 0, sitemap pages depth 1
Main content only	on	Mozilla Readability keeps the main text. Off converts the whole body without header, footer and navigation
Content selector	none	CSS selector for the content to convert, such as article.post. Pages where it matches nothing use the normal extraction
Remove elements	none	CSS selectors to drop before conversion, such as .ads, #comments
Keep links	on	Links keep absolute URLs. Off keeps only the link text
Keep images	off	Images as Markdown image links
Include linked PDF files	off	Also convert in-scope linked PDFs to Markdown (text layer, no OCR). Up to 20 MB and 300 pages per file. Each PDF counts as one page
Split into chunks for RAG	on	Adds the chunks list
Chunk size and overlap	500 and 0 tokens	Largest chunk and the overlap between chunks, in estimated tokens
Save llms.txt and llms-full.txt	on	Saves both files to the key-value store
Change-only mode and state key	off, "default"	Save only new or changed pages since the last change-only run with the same key
Modified since	none	Load only pages changed on or after a date, such as 2026-10-01 or 30 days. Uses sitemap dates and the If-Modified-Since header. Skipped pages are free
Parallel requests	3	Pages loaded at the same time
Requests per minute	120	Rate cap. Requests to one site are spaced evenly
Proxy	none	Optional

Example input

{
  "startUrls": [
    {
      "url": "https://crawlee.dev/js/docs/quick-start"
    }
  ],
  "maxPages": 100,
  "scope": "path",
  "useSitemaps": true,
  "chunking": true,
  "chunkSize": 500,
  "saveLlmsTxt": true
}

Example output

One page, with the Markdown and chunk text shortened:

{
  "id": "https://crawlee.dev/js/docs/quick-start",
  "url": "https://crawlee.dev/js/docs/quick-start",
  "loadedUrl": "https://crawlee.dev/js/docs/quick-start",
  "title": "Quick Start",
  "description": "With this short tutorial you can start scraping with Crawlee in a minute or two. To learn more, read the Introduction.",
  "language": "en",
  "markdown": "Version: 3.18\n\n# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two. To learn in-depth how Crawlee works, read the [Introduction](https://crawlee.dev/js/docs/introduction), which is a comprehensive step-by-step guide for creating your first scraper.\n\n## Choose your crawler\n\nCrawlee ...",
  "markdownTruncated": false,
  "wordCount": 575,
  "tokenEstimate": 4346,
  "contentHash": "b1c42a2d0364964c01af5aebb5bed6c0",
  "changeStatus": null,
  "depth": 0,
  "discoveredBy": "start",
  "sitemapLastmod": null,
  "httpStatus": 200,
  "chunkCount": 20,
  "chunks": [
    {
      "index": 0,
      "headingPath": [
        "Quick Start"
      ],
      "text": "Version: 3.18\n\n# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two. To learn in-depth how Crawlee works, read the ...",
      "tokenEstimate": 73,
      "charCount": 291
    },
    {
      "index": 1,
      "headingPath": [
        "Quick Start",
        "Choose your crawler"
      ],
      "text": "## Choose your crawler\n\nCrawlee comes with three main crawler classes: [`CheerioCrawler`](https://crawlee.dev/js/api/cheerio-crawler/class/CheerioCrawler), ...",
      "tokenEstimate": 109,
      "charCount": 433
    }
  ],
  "scrapedAt": "2026-10-22T08:58:58.488Z"
}

Token estimate

tokenEstimate counts about 4 characters per token for English and other Latin text, and 2 characters per token for other scripts such as Arabic or Chinese. It is an estimate. Exact counts depend on your model's tokenizer.

Polite crawling

• robots.txt is read and followed for every page and every sitemap.
• A Crawl-delay in robots.txt sets one request at a time with that delay between requests (up to 30 seconds).
• Requests to one site are spaced evenly. The default is 2 requests per second at most.
• Each request has a 45-second timeout and at most 2 retries.

Pages that are skipped

This Actor never bypasses bot protection, CAPTCHAs or logins. It skips:

• pages with a bot challenge or CAPTCHA wall (for example Cloudflare, DataDome or PerimeterX),
• pages that answer HTTP 401, 403 or 429,
• login pages (a password form with little other text),
• files that are not HTML, such as images, and PDFs unless Include linked PDF files is on,
• pages larger than 5 MB, and redirects that leave the crawl scope.

Skipped pages are logged and not charged.

Pricing

Pay per event. "Page unchanged" applies only in change-only mode. The default run saves up to 30 pages.

Event	Free plan	GOLD plan and higher
Run start	$0.005	$0.005
Page conversion	$0.0015 ($1.50 per 1,000)	$0.0012 ($1.20 per 1,000)
Unchanged page checked	$0.001 ($1.00 per 1,000)	$0.001 ($1.00 per 1,000)

Pages in one run	Free plan	GOLD plan and higher
100	$0.155	$0.125
1,000	$1.505	$1.205

Costs include the run start fee. The SILVER plan pays a price between the two columns.

You can set a maximum cost per run in the run options. The crawl stops when it reaches it.

Change-only mode

Turn on Change-only mode and schedule the Actor. The first run saves every page as "new". Later runs with the same state key compare each page's Markdown hash with the last run. Only new and changed pages are saved, with changeStatus set. Unchanged pages cost the lower "Page unchanged" price in the table above. The crawler still loads and converts them. llms.txt and llms-full.txt always list every page converted in the run. Use a different state key for each site you watch.

Data use

You are responsible for the sites you crawl. Crawl sites you own, or sites whose terms allow reuse of their content. Web pages can hold personal data, such as names in a team page. Handle it under GDPR and other laws that apply to you.

Limits

• No JavaScript rendering. Pages built by scripts give little text; such rows get contentWarning: "needs-javascript".
• Markdown is cut at 500,000 characters per page (markdownTruncated is true).
• Sitemaps: up to 50 sitemap files and 90 seconds of reading. With a page limit, the crawl reads up to twice that many sitemap URLs.

FAQ

Can it crawl a whole site? Yes. Set the scope to "Same domain" and Maximum pages to 0.

Does it work with LangChain, LlamaIndex or a vector database? Yes. Each chunk has its text and heading path. Load the dataset with the Apify integration for your tool, or download it as JSON.

How do I remove ads, cookie notes or comments from the Markdown? Put their CSS selectors in Remove elements, for example .ads, .cookie-notice, #comments. They are dropped before conversion.

How do I convert only one part of each page? Put a CSS selector in Content selector, for example main .article-body. Only the matched elements become Markdown. Check contentSelectorMatched to see where it matched.

Why is a page short or empty? The page probably builds its content with JavaScript. This crawler reads the HTML the server sends.

Related Actors

• PDF to Markdown and Text Converter: turn PDF files into Markdown and text, with OCR for scans.
• Website Screenshot & PDF Capture: full-page screenshots and PDFs of pages that need a browser.
• HTTP Request Runner: call many URLs or APIs in bulk and keep the raw response.
• SEO Audit & Broken Link Checker: check the same site for broken links, redirects and missing tags.