Website to Markdown: Fast Crawler for LLMs, RAG & AI Agents

Turn any website, docs site, or blog into clean Markdown, one row per page, ready for an LLM, a vector database, or an AI agent. Only the main content is kept: menus, headers, footers, sidebars, and cookie banners are stripped. Headings, lists, tables, code blocks, and links come through as proper Markdown.

• Fast and cheap: plain HTTP with no browser. 8 docs pages convert in about 2 seconds.
• Stays in scope: start at https://docs.example.com/v2/ and it only crawls pages under /v2/ on that site. It uses the sitemap when there is one and follows links otherwise.
• Clean Markdown: <main>/<article> detection, ARIA landmarks and boilerplate classes removed, nested lists and tables kept, code fences preserved.
• Polite: obeys robots.txt and makes at most 4 requests at a time.

Sample output (real, docs.python.org, Oct 2026)

{
  "type": "page",
  "startUrl": "https://docs.python.org/3/library/json.html",
  "url": "https://docs.python.org/3/library/marshal.html",
  "loadedUrl": "https://docs.python.org/3/library/marshal.html",
  "title": "marshal — Internal Python object serialization — Python 3.15.2 documentation",
  "description": "This module contains functions that can read and write Python values in a binary format. …",
  "language": "en",
  "markdown": "# `marshal` — Internal Python object serialization\n\n---\n\nThis module contains functions that can read and write Python values in a binary format. The format is specific to Python, but independent of machine architecture issues … [shortened]",
  "wordCount": 995,
  "crawledAt": "2026-10-30T20:53:53+00:00"
}

That run converted 8 pages under /3/library/ (json, pickle, decimal, mailbox…, 339 to 10,439 words each) in about 2 seconds.

How to use

1. Add start URLs: a page, a docs section, or a homepage.
2. Set max pages per start URL (1 = just that page).
3. Optional: only / skip URLs containing, follow links, use sitemap, main content only, keep links / images.
4. Run, then export (JSON, CSV, Excel), or read the dataset from your RAG pipeline through the API, LangChain, or LlamaIndex.

Sample input

{ "startUrls": ["https://docs.python.org/3/library/"], "maxPagesPerStartUrl": 50, "excludeUrlsContaining": ["/whatsnew/"] }

Pricing

• $0.002 per page converted ($2 per 1,000 pages). No start fee.
• Pages that fail, aren't HTML, or are disallowed by robots.txt are free.

Who it's for

• RAG and chatbot builders: load docs, help centers, and blogs into a vector store
• AI agents: give an agent clean page text instead of raw HTML
• Content teams: a Markdown copy of a site for migration, audits, or summaries
• Researchers: clean text from many pages for analysis

Works well with

• Sitemap URL Extractor: list every URL first, then convert the ones you want
• RSS Feed Reader: watch for new posts, then convert them

FAQ

Does it run JavaScript? No. That's why it's fast and cheap. Sites that build their pages entirely in JavaScript come back thin. For those, use a browser-based crawler. PDFs? Skipped (free). HTML pages only. Will it crawl the whole internet from my start URL? No. It stays on the same site and under the start URL's folder.

More tools from the same studio: Kramer Labs.

Data source and responsible use

It reads public web pages with ordinary requests, obeys each site's robots.txt (RFC 9309), and never logs in or bypasses blocks. Page content belongs to its owners: use it for indexing, analysis, and AI retrieval in line with each site's terms and copyright, and don't republish it as your own.