Website to Markdown Crawler - RAG-Ready Chunks for LLMs

Crawl docs sites, blogs, and help centers and convert every page into clean Markdown with its URL, title, and meta description. Get one row per page, or switch on chunking to get overlapping, embedding-ready chunks for RAG pipelines, vector databases, chatbots, and AI agents. Fast HTTP crawler, no AI API key needed. $2.00 per 1,000 rows.

Real output from a default run of Website to Markdown Crawler in Apify Console.

What can this website to Markdown crawler do?

• Convert web pages to Markdown - headings, lists, links, tables, and code blocks preserved.
• Remove boilerplate - navigation, headers, footers, cookie banners, scripts, and similar clutter are stripped; the densest main-content container (article, main, #content...) is preferred.
• Crawl a whole site or one section - set crawl depth, page limit, same-domain restriction, and include/exclude URL globs.
• Chunk for RAG - paragraph-aware chunks with configurable size, overlap, and minimum length, each tagged with chunkIndex and chunkCount.
• Keep source attribution - every row carries the page URL and title, so your chatbot can cite sources.
• Optional cleaned HTML of the main content alongside the Markdown.

What data can I extract?

Field	Description
url	Source page URL
title	Page title
description	Meta description
content	Markdown of the page (or of one chunk)
contentChars	Length of content
chunkIndex, chunkCount	Position of the chunk within its page (0 / 1 when chunking is off)
html	Cleaned main-content HTML (only when saveHtml is on)
depth	Link hops from the start URL
crawledAt	Timestamp

The key-value store SUMMARY record reports pages crawled, pages failed, total chunks, total characters, and average chunk size.

How much does it cost?

Pay-per-event: $0.002 per dataset row plus a tiny Actor-start fee.

• Chunking off: one row per page = $2.00 per 1,000 pages.
• Chunking on: one row per chunk. A typical docs page produces a few chunks at the default 1,000 characters, so cost per page rises accordingly. Increase chunkSize to reduce rows, or turn chunking off and chunk in your own pipeline.

Check the Pricing tab for current rates.

How to convert a website to Markdown

1. Add one or more start URLs (for example https://docs.example.com/).
2. Set maxCrawlPages and maxCrawlDepth (0 = only the start URLs).
3. Optionally limit the crawl with includeUrlGlobs such as https://docs.example.com/guides/*.
4. Choose whole pages (chunkContent: false) or RAG chunks (chunkContent: true).
5. Click Start and export JSON, JSONL, or CSV - or load the dataset straight into LangChain or LlamaIndex.

Input example

{
  "startUrls": ["https://docs.apify.com/"],
  "maxCrawlPages": 200,
  "maxCrawlDepth": 2,
  "sameDomainOnly": true,
  "includeUrlGlobs": ["https://docs.apify.com/platform/*"],
  "chunkContent": true,
  "chunkSize": 1500,
  "chunkOverlap": 150
}

Output example

{
  "url": "https://docs.apify.com/platform/actors",
  "title": "Actors | Platform | Apify Documentation",
  "description": "Learn how to develop, run and share serverless cloud programs.",
  "chunkIndex": 0,
  "chunkCount": 3,
  "content": "# Actors\n\nActors are serverless cloud programs that can perform anything from a simple action...",
  "contentChars": 1432,
  "depth": 1,
  "crawledAt": "2026-10-25T10:00:00.000Z"
}

Illustrative output shape.

Use cases

• RAG knowledge bases - index your docs or help center for a support chatbot.
• Custom GPTs and AI agents - give an agent a clean, cited copy of a website.
• Vector database ingestion - Pinecone, Qdrant, Weaviate, pgvector, Chroma.
• Content migration - move a blog or docs site to a Markdown-based CMS.
• Competitive research - read a competitor's docs or pricing pages as plain Markdown.

Integrations and API

Use the Apify API, LangChain and LlamaIndex Apify loaders, Make, Zapier, n8n, or the Apify MCP server.

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("groupoject/ai-rag-web-crawler").call(run_input={
    "startUrls": ["https://docs.example.com/"],
    "maxCrawlPages": 100,
    "chunkContent": True,
})
docs = [(row["url"], row["content"]) for row in client.dataset(run["defaultDatasetId"]).iterate_items()]

FAQ

Does it render JavaScript? No. It is an HTTP crawler (Cheerio) built for server-rendered sites such as most docs platforms, blogs, and help centers. Single-page apps that load content with JavaScript may return little text.

Does it need an OpenAI or other AI key? No. Extraction and chunking are deterministic.

How is this different from Apify's Website Content Crawler? Website Content Crawler is a general-purpose crawler with browser rendering and many options. This Actor is a lightweight HTTP-only crawler with built-in chunking and a simple per-row price.

Can it get blocked? Some sites block datacenter IPs; enable Apify Proxy in proxyConfiguration.

Related Actors

• Technical SEO Audit Crawler - audit the same pages for SEO issues.
• Contact Details Scraper - extract emails, phones, and tech stack from company sites.