Website to Markdown Crawler: LLM and RAG Ready Markdown

Website to Markdown Crawler for LLMs, RAG and AI Agents

Crawl a website from one or more start URLs and get clean Markdown for every page, with title and word count, ready for AI agents, RAG pipelines, vector databases and LLM context. Turn docs sites, help centers and blogs into Markdown in one run. An alternative to the Website Content Crawler with flat, predictable per-page pricing instead of compute-based billing.

What it does

• Follows links only within the same domain as each start URL (optionally only under the start path).
• Respects robots.txt, including crawl-delay.
• Stops at your max-pages limit, so the cost of a run is known in advance.
• Strips navigation, headers, footers, scripts and forms, then converts the main content to Markdown.
• Optional JavaScript rendering with Playwright (off by default to keep runs cheap and fast).

Input

See .actor/input_schema.json. Example:

{
  "startUrls": ["https://docs.apify.com/academy"],
  "maxPages": 5,
  "stayUnderStartPath": true
}

Output

One dataset item per page:

{ "url": "...", "title": "...", "markdown": "# Heading ...", "wordCount": 812, "crawledAt": "2026-10-15T17:00:00.000Z" }

Pricing

Flat per-page pricing: $1.50 per 1,000 pages, or $3 per 1,000 pages with renderJs on. Error rows are not charged.

FAQ

Does it crawl other domains? No. Links are followed only within the same domain as each start URL, and with stayUnderStartPath only under the start path (for example just /docs/).

Does it handle JavaScript sites? Yes, if you set renderJs to true. Pages then load in headless Chromium via Playwright. It is off by default to keep runs cheap and fast.

How do I control cost and run size? Set maxPages (default 50, maximum 10000). The run stops when that many pages are saved across all start URLs.

Can an AI agent or MCP client use it? Yes. Start URLs in, one JSON record per page with a markdown field out, via the Apify API or an MCP client that runs Apify Actors.

How is it priced? $1.50 per 1,000 pages, or $3 per 1,000 pages with renderJs on, billed per successful result. Error rows are not charged.

Local test

npm install
mkdir -p storage/key_value_stores/default && cp test-input.json storage/key_value_stores/default/INPUT.json
npm start

About

Built and maintained by Small Gears Co (https://smallgearsco.com). Questions or feature requests: contact@smallgearsco.com