Website to Markdown for AI & RAG – Fast Crawler + llms.txt Generator

Turn any website, documentation portal, blog or list of URLs into clean, LLM-ready Markdown. Menus, footers, ads and cookie banners are removed; headings, lists, tables and code blocks are kept. Perfect input for RAG pipelines, custom GPTs, chatbots, vector databases (Pinecone, Qdrant, Weaviate, pgvector) and AI agents.

Why this Actor

• Clean Markdown, not HTML soup. Main-content extraction (Readability) + GitHub-flavoured Markdown tables and fenced code.
• Fast & light. Plain HTTP fetching, no headless browser – hundreds of pages per minute.
• Simple, predictable price. You pay a small fixed price per converted page. Failed pages and pages skipped by your filters are free.
• Sitemap + link crawling. Finds pages via sitemap.xml and on-page links, stays on the same domain and section.
• llms.txt built in. Get an llms.txt index and a single llms-full.txt file of the whole site – drop it straight into ChatGPT, Claude or your agent.
• Rich metadata: title, description, language, author, published date, canonical URL, word count.

Use cases

• Build a support chatbot from your help center or docs.
• Feed competitor websites or industry blogs into your knowledge base.
• Generate llms.txt for your own site (AI-search / GEO optimization).
• Create fine-tuning or evaluation datasets from public documentation.
• Give an AI agent (LangChain, LlamaIndex, n8n, Make, Zapier, MCP) a "read this website" tool.

Quick start

1. Paste one or more Start URLs (e.g. https://docs.example.com).
2. Set Max pages. Turn Crawl off if you only want the exact URLs you listed.
3. Run, then download the dataset (JSON, CSV, Excel) or open llms-full.txt from the Output tab.

Input example

{
  "startUrls": ["https://docs.apify.com/academy"],
  "crawl": true,
  "maxPages": 200,
  "maxDepth": 3,
  "excludePatterns": ["*/changelog/*"],
  "contentMode": "main",
  "buildLlmsTxt": true
}

Output example

{
  "url": "https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/name-your-actor",
  "title": "Name your Actor | Academy | Apify Documentation",
  "description": "Apify's standards for Actor naming…",
  "language": "en",
  "wordCount": 1236,
  "markdown": "## Name your Actor\n\nApify's standards for Actor naming…",
  "crawledAt": "2026-10-14T10:20:00.000Z"
}

Extra outputs in the key-value store: llms.txt, llms-full.txt, and FAILED_URLS (pages that could not be fetched).

Integrations

Use the Apify API, or connect to LangChain (ApifyDatasetLoader), LlamaIndex, Pinecone / Qdrant integrations, n8n, Make or Zapier. Each item's markdown field is ready to chunk and embed.

Good to know

• Works best on server-rendered sites (docs, blogs, marketing sites, news, wikis). Pages that render content only with JavaScript may return little text.
• robots.txt is respected by default.
• If a site blocks requests, enable a proxy in the input.
• Only crawl content you have the right to use; respect each website's terms.

FAQ

How is this different from a full browser crawler? It's much faster and cheaper for the 90% of sites that don't need a browser, and the price is fixed per page with no surprise compute bills.

Can I convert only specific pages? Yes – turn Crawl off and list the URLs.

Found a page that converts badly? Open an issue with the URL and we'll improve extraction.