Website to Markdown for AI/LLM — Cheap RAG Crawler

Turn any website into clean, LLM-ready Markdown. The Actor crawls pages over plain HTTP (no browser), strips navigation, footers, cookie banners and ads, and returns Markdown (plus optional plain text, cleaned HTML and RAG chunks) with metadata. Because it never starts a browser, it is fast and costs US$0.50 per 1,000 pages.

What you get per page

• markdown — main content as clean Markdown (headings, lists, tables, fenced code with language, links, optional images)
• text / html — optional plain text and cleaned HTML
• title, description, language, canonicalUrl
• headings — outline with level and text
• wordCount, textLength, linksCount, depth, statusCode, fetchedAt
• chunks[] — optional RAG chunks (index, text, charStart, charEnd, heading) with configurable size and overlap
• needsBrowser — true when the page is an empty JavaScript shell (single-page app). These pages are returned (not failed) and not charged, so you know which URLs need a browser-based crawler.

Features

• Single pages or same-domain crawl with maxDepth and maxItems (default 20 pages)
• Include / exclude URL patterns (https://example.com/docs/**)
• sitemap.xml discovery (from robots.txt or /sitemap.xml, sitemap indexes and .gz supported)
• Respects robots.txt by default (switch off if you own the site)
• Readability-style main-content detection; layout tables unwrapped; real tables kept as Markdown tables
• Handles non-UTF-8 pages (charset detection), redirects, canonical-URL de-duplication
• Automatic proxy escalation: datacenter first, residential only when a site blocks you

Input example

{
  "startUrls": [{ "url": "https://docs.python.org/3/tutorial/index.html" }],
  "crawlMode": "crawl",
  "maxDepth": 1,
  "maxItems": 20,
  "outputFormats": ["markdown"],
  "enableChunking": true,
  "chunkSize": 1000,
  "chunkOverlap": 100
}

Output example

{
  "url": "https://docs.python.org/3/tutorial/index.html",
  "canonicalUrl": null,
  "title": "The Python Tutorial — Python 3.15.0 documentation",
  "description": "Python is an easy to learn, powerful programming language...",
  "language": "en",
  "markdown": "# The Python Tutorial\n\nPython is an easy to learn...",
  "text": null,
  "html": null,
  "textLength": 6283,
  "wordCount": 1067,
  "headings": [{ "level": 1, "text": "The Python Tutorial" }],
  "linksCount": 174,
  "depth": 0,
  "needsBrowser": false,
  "statusCode": 200,
  "fetchedAt": "2026-10-13T10:00:00.000Z",
  "chunks": [{ "index": 0, "text": "# The Python Tutorial ...", "charStart": 0, "charEnd": 996, "heading": "The Python Tutorial" }],
  "chunksCount": 18,
  "truncated": false,
  "error": null
}

Pricing

Event	Price	When
page-result	US$0.0005 (US$0.50 / 1,000)	per page converted and saved
Actor start	US$0.00005	once per run

Pages that need a browser (needsBrowser: true), failed pages and robots-blocked pages are not charged.

FAQ

Does it run JavaScript? No. It reads the HTML the server sends, which covers docs sites, blogs, news, wikis, shops' content pages and most marketing sites. Pure single-page apps come back flagged needsBrowser: true.

How do I feed it to an LLM or vector DB? Use markdown, or enable chunking and embed each chunks[].text; heading gives you the section for metadata.

What about huge pages? Fields are capped at 1.5 million characters per page, and pages with more than about 3 MB of HTML per GB of run memory are converted up to that size (links further down are still followed); truncated tells you when either happened. Responses over 8 MB are skipped.

Is it allowed to crawl a site? Check the site's terms and robots.txt. The Actor respects robots.txt by default.

Use cases: RAG knowledge bases from documentation, LLM training/eval corpora, site migration to Markdown, content audits, changelog and blog monitoring.

En español

Convierte sitios web en Markdown limpio para IA/RAG usando solo HTTP (sin navegador): quita menús, pies de página y banners de cookies; entrega título, descripción, idioma, esquema de encabezados, conteo de palabras y, opcionalmente, trozos (chunks) con solapamiento para embeddings. Soporta crawl del mismo dominio, sitemap.xml y robots.txt. Las páginas que solo funcionan con JavaScript se marcan needsBrowser: true y no se cobran. Precio: US$0,50 por 1.000 páginas.