Website to Markdown Crawler for LLM, RAG and AI Agents

Use it to crawl a website into clean Markdown for LLM context, RAG pipelines, vector databases and AI agents.

Website to Markdown Scraper crawls a website and converts the main content of each page into clean Markdown. Use it to feed LLMs, RAG pipelines, vector databases, custom GPTs and AI agents, or to mirror documentation as Markdown.

Why this Actor

• Clean main content. It prefers <main>/<article> and strips site chrome.
• Polite by design. It obeys robots.txt and Crawl-delay, waits at least 1 second between requests, and identifies itself with a clear user agent.
• Same-site crawl with maxPages, maxDepth and an optional URL includePattern.
• Pay only for pages saved. There is no subscription.

Example run

Real input and the first output record from run WkwarA6LKhStwlqcd on 2026-10-30. Long text fields are shortened here with "...".

Input:

{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy"
    }
  ],
  "maxPages": 20,
  "maxDepth": 2
}

Output (first record):

{
  "url": "https://docs.apify.com/academy",
  "title": "Apify Academy | Academy | Apify Documentation",
  "description": "Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.",
  "lang": "en",
  "markdown": "# Apify Academy\n\nLearn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.\n\n## Beginner courses\n\n[\n\n![](/img/academy/scraping-basics-javascript.svg)\n\n#### Web scraping basi...",
  "wordCount": 207,
  "depth": 0
}

Input

Field	Description
startUrls	One or more URLs to start from
maxPages	Stop after this many pages
maxDepth	Link depth from the start URLs (0 = start pages only)
includePattern	Optional regex; only matching URLs are crawled

Output (one item per page)

{"url":"https://docs.example.com/guide","title":"Guide","description":"...","lang":"en","markdown":"# Guide\n\n...","wordCount":648}

Export as JSON, CSV, Excel or via API, or connect to LangChain, LlamaIndex, Make, Zapier or n8n through Apify integrations.

Limits and fair use

It reads only public HTML pages and never logs in. Pages disallowed by robots.txt are skipped and counted. You are responsible for respecting each site's terms and copyright.

Pricing

Pay-per-event: a small start fee plus a fee per page saved (see the Pricing tab).