Website to Markdown for AI & RAG: Fast Crawler

Crawl any website or documentation section and convert every page into clean, LLM-ready Markdown:

• main content only, without menus, cookie banners or footers
• page metadata, headings and last-modified dates
• optional chunks with overlap, ready for embeddings

Built for RAG pipelines, AI agents, chatbots, knowledge bases and fine-tuning datasets.

Why this crawler

• Fast and cheap. Plain HTTP, no headless browser: 50 documentation pages in about 7 seconds in testing. You pay per converted page, and thin or empty pages below your word threshold are free.
• Clean Markdown.
  • Mozilla Readability main-content extraction, GitHub-flavored Markdown tables and fenced code blocks.
  • Absolute links, heading-anchor noise removed.
• Finds the pages that matter.
  • Discovers URLs from robots.txt sitemaps and follows internal links.
  • Stays inside the section you started from (e.g. /docs/), with include/exclude URL globs.
• RAG-ready output.
  • Heading-aware chunking with configurable size and overlap, and token estimates per page and chunk.
  • lastModified from sitemaps for incremental re-indexing.
• Polite by default. Respects robots.txt and removes tracking parameters.

Input example

{
  "startUrls": [{ "url": "https://docs.apify.com/platform/actors" }],
  "maxPages": 500,
  "maxCrawlDepth": 5,
  "stayWithinStartPath": true,
  "useSitemaps": true,
  "excludeUrlGlobs": ["**/changelog/**"],
  "extractionMode": "article",
  "includeLinks": true,
  "chunkSize": 1500,
  "chunkOverlap": 200
}

Output example

{
  "url": "https://docs.apify.com/get-started",
  "title": "Get started | Platform | Apify Documentation",
  "description": "Learn how the Apify platform works - what Actors are, how they get to Apify Store, and how to start by running existing Actors or building your own.",
  "language": "en",
  "canonicalUrl": "https://docs.apify.com/get-started",
  "lastModified": "Thu, 17 Sep 2026 12:02:20 GMT",
  "markdown": "Apify is a cloud platform for web scraping, data extraction, and automation. Everything on it revolves around *Actors* - serverless programs that run in the cloud. This page explains how the platform fits together and where to start, whether you want to use ex…",
  "headings": [
    {
      "level": 1,
      "text": "Get started"
    },
    {
      "level": 2,
      "text": "How Apify works"
    },
    {
      "level": 2,
      "text": "How Actors get to Apify Store"
    }
  ],
  "wordCount": 388,
  "tokenEstimate": 721,
  "chunks": [
    {
      "index": 0,
      "text": "Apify is a cloud platform for web scraping, data extraction, and automation. Everything on it revolves around *Actors* -…",
      "tokenEstimate": 293
    },
    {
      "index": 1,
      "text": "rom your code like a regular API.\n\nFor how the platform is built, what happens during a run, and how workloads stay isol…",
      "tokenEstimate": 238
    }
  ],
  "crawlDepth": 1
}

Use cases

• RAG & chatbots: index product docs, help centers and blogs into vector databases.
• AI agents: give agents up-to-date website knowledge as Markdown.
• Fine-tuning & evaluation datasets: collect clean text corpora from websites.
• Content migration & SEO audits: export site content with headings and metadata.
• Change monitoring: re-crawl with sitemap lastModified dates.

Pricing

Pay per converted page, plus a negligible start fee per run. Pages shorter than Skip pages shorter than (words) are not returned or charged.

Notes

• Pages that render their content only with client-side JavaScript may produce little text. For those, a browser-based crawler is needed.
• Respect each website's terms of use when crawling and reusing content.

Related Actors

Other scrapers from the same maker, built the same way — no browser where it isn't needed, one flat price per result:

• Google Ads Transparency Scraper
• Website Contact & Email Scraper