Website to Markdown Crawler for LLMs, RAG and AI agents ($1 per 1,000 pages)

Crawl any website and get every page's main content as clean Markdown: no menus, footers, cookie banners or scripts, with code blocks kept (language and indentation), tables, links and images as full URLs, plus each page's title, description, language, dates and headings.

Turn on chunks and each page is split on its headings into ready-to-embed pieces with their heading path, for RAG, vector databases, custom GPTs and AI agents. It reads the site's sitemaps, opens JavaScript pages in Chrome only when needed, and respects robots.txt.

What you get

Field	What it holds
Page rows	url, title, description, language, canonicalUrl, author, publishedAt, modifiedAt, imageUrl, depth, referrer, statusCode, headings, wordCount, markdown; optional text, html
Chunks (optional)	chunks inside each page, or one row per chunk: chunkIndex, headings, text, characters, chunkCount
SUMMARY	pages delivered, failed, left out by robots.txt, duplicates, pages found in sitemaps, pages opened in Chrome, what was charged

What people use it for

• RAG and vector search: a docs site or help centre as chunks to embed
• Custom GPTs and assistants: your whole website as knowledge files
• AI agents: read a site's pages as Markdown in one call
• Content audits and migrations: every page's text, headings and metadata

How to use it

1. Add a start URL: it crawls that site under that path (start at /docs/ for the docs only).
2. Set Most pages and, for RAG, Chunks.
3. Click Start and download the pages as JSON, CSV or Excel, or read them by API.

Input

Field	Example	Notes
Start URLs	https://docs.example.com/guide/	Crawls under this path
Most pages	100	Up to 100,000
Use the site's sitemaps	on	Finds pages no link points to
JavaScript pages	auto	Chrome only when the HTML is empty
Chunks for RAG	none	per-page or rows
Chunk size	2000	Characters, overlap 200
Only / skip URLs matching	*/blog/*	Wildcards
Content selector	article	Optional: the part to keep

Output

Each result is one row in the run's dataset (download it as JSON, CSV, Excel, XML or HTML, or read it through the API). An example row (values shortened):

{
  "url": "https://docs.example.com/tutorial/controlflow.html",
  "title": "4. More Control Flow Tools",
  "description": "As well as the while statement just introduced, the language uses a few more…",
  "language": "en",
  "depth": 1,
  "headings": [
    "4. More Control Flow Tools",
    "4.1. `if` Statements",
    "4.2. `for` Statements"
  ],
  "wordCount": 5915,
  "markdown": "# 4. More Control Flow Tools\n\nAs well as the [`while`](https://docs.example.com/reference/compound_stmts.html#while) statement…\n\n## 4.1. `if` Statements\n\n```python\n>>> x = int(input(\"Please enter an integer: \"))\n>>> if x < 0:\n...     x = 0\n```…",
  "chunks": [
    {
      "chunkIndex": 0,
      "headings": [
        "4. More Control Flow Tools"
      ],
      "text": "# 4. More Control Flow Tools…",
      "characters": 1475
    }
  ],
  "crawledAt": "2026-10-17T18:45:15Z"
}

From runs on 17 October 2026 (a programming language's documentation and a company blog, addresses replaced): 12 tutorial pages in 7 seconds with every code example fenced and indented and the navigation and heading anchors removed; on the blog, the crawl stayed under /blog, found 24 posts in its sitemaps and wrote 8 of them as 33 chunks. A record named SUMMARY in the run's key-value store counts every page's outcome and what was charged.

Pricing

• $0.001 per page delivered with its content ($1 per 1,000 pages); chunks cost nothing extra
• Free: pages that fail, are blocked, are left out by robots.txt or your patterns, or repeat another page
• No AI usage: the conversion runs without a language model

What typical jobs cost at the list price:

Job	Cost
A 200-page docs site	$0.20
A blog's 50 latest posts as chunks	$0.05
10,000 pages	$10

On Apify's Scale and Business plans the per-event prices are lower (Store discounts). Set Maximum cost per run and the Actor stops cleanly when it is reached: you never pay more than that.

Tips

• Start at the section you need (/docs/, /help/) and the crawl stays inside it; switch off Stay under the start URL's path for the whole site.
• Choose one row per chunk to load the dataset straight into a vector database.
• Use Skip URLs matching for tag, search and login pages (*/tag/*, *?page=*).
• Schedule it weekly to keep an AI's knowledge of a site fresh.

Use it through the API

Python

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("zaiq/llm-website-crawler").call(run_input={"startUrls": [{"url": "https://docs.example.com/"}], "maxPages": 200, "chunking": "rows"})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row)

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('zaiq/llm-website-crawler').call({"startUrls": [{"url": "https://docs.example.com/"}], "maxPages": 200, "chunking": "rows"});
const { items } = await client.dataset(run.defaultDatasetId).listItems();

HTTP (one call, results back)

curl -X POST 'https://api.apify.com/v2/acts/zaiq~llm-website-crawler/run-sync-get-dataset-items?token=<YOUR_APIFY_TOKEN>' \
  -H 'Content-Type: application/json' \
  -d '{"startUrls": [{"url": "https://docs.example.com/"}], "maxPages": 200, "chunking": "rows"}'

Integrations and schedules

• Schedules: run it every hour, day or week from the Apify Console (Schedules) and keep a history of results.
• Make, Zapier, n8n, Google Sheets, Slack, Airtable: connect it through Apify's integrations, or send each finished run to a webhook.
• Datasets: results stay in the run's dataset; export them or read them by API whenever you need them.

Use it with AI agents (MCP)

Add Apify's MCP server (https://mcp.apify.com) to Claude, ChatGPT, Cursor, VS Code or any MCP client. The agent can find this Actor and run it for you: ask it for "read the docs at this address and give me every page as Markdown". AI agents can also pay for runs directly through Apify's agentic payments.

Limits

• It connects directly and through Apify's datacenter proxies: a site that blocks both is reported as failed (free). The Website Content Crawler to Markdown by Country reads such sites through residential connections.
• PDFs, images and other files are skipped.
• Pages behind a login are not read.
• A page's Markdown is cut at 400,000 characters.

FAQ

Is it an alternative to Apify's Website Content Crawler?

Yes, with a fixed price per page instead of compute time, chunks on headings and code blocks kept with their language.

Does it read JavaScript sites?

Yes: a page whose HTML carries no content opens in Chrome (auto), or every page with Always.

Does it follow robots.txt?

Yes by default; it can be switched off for your own sites.

Can I get plain text or HTML too?

Yes: switch on Also plain text or Also the page's HTML.

Related Actors from Zaiq

• ai-web-scraper: the data you describe from any page, as JSON
• website-crawler-by-country: sites read from a chosen country through residential connections
• bulk-ai-runner: an AI prompt on every page
• company-research-agent: company profiles from websites

Support

Questions, a site that changed, or a field you need? Open an issue on this Actor's Issues tab: we answer there, usually within a day.

---

Also found as: website content crawler, website to markdown, rag crawler, llm crawler, web crawler for ai, docs to markdown, crawl website for chatgpt, markdown scraper, vector database ingestion.