Website to Markdown Crawler - LLM & RAG Ready Content

Crawl documentation sites, blogs and knowledge bases into clean Markdown for LLMs, RAG pipelines and AI assistants. Only the main content is kept (menus, footers and cookie bars are dropped); headings, lists, tables and links are preserved. Optional chunking for vector databases.

$0.0018 per page converted, no start fee, and a hard page limit per site, so you always know the maximum cost before you start (maxPagesPerSite × $0.0018). Pages that fail, are not HTML, are empty or are disallowed by robots.txt are free.

Quick start: example input, output and cost

Turn docs sites and blogs into LLM-ready Markdown with chunks. Hard page limit so the cost is known up front, robots.txt respected, failed pages free.

Example input

{
  "startUrls": [
    "https://www.rfc-editor.org/rfc/rfc9309.html"
  ]
}

Cost example: A 300-page docs site x $0.0018 = $0.54 (the page limit caps it).

Why this one

• Predictable cost. Fixed price per page plus a hard limit, so a run cannot surprise you with a large bill.
• Fast and light. Plain HTTP crawling (no browser), so it runs in 512 MB and does not run out of memory in n8n/Make flows. Sites that need JavaScript to show content are out of scope: use a browser-based crawler for those.
• Deterministic. Breadth-first crawl, same input = same pages; stayWithinPath, maxDepth, include/exclude regex, optional sitemap seeding.
• One bad site does not stop the run. A slow or broken site is skipped and the others carry on; every skipped URL is listed with a reason in ERRORS.
• Privacy by default. Email addresses and phone numbers are replaced with [redacted] (switch off with redactContactDetails: false for your own sites). Social networks and profile sites are not crawled.
• PDFs? Use our Document to Markdown OCR actor for linked PDF files.

Keep a RAG index in sync, pay only for what changed

Set syncName (for example docs-sync) and schedule the actor. The first run returns every page with syncStatus: "new". Later runs with the same name:

• return and charge only pages that are new or whose Markdown changed (syncStatus: "changed"),
• skip pages whose content is the same as last time, free of charge,
• write a SYNC_REPORT record per start URL with counts of new, changed and unchanged pages and the list of removed URLs (pages that now return 404/410 or are no longer linked), so you can delete their vectors.

Removed pages are only reported when the crawl reached every page it found; if the page limit or the run's cost limit cut it short, removedChecked is false. A page that fails for another reason (timeout, 5xx) keeps its stored state and is not reported as removed.

Example: a 400-page docs site where 12 pages change per week costs $0.72 for the first run and about $0.02 per weekly sync after that.

Each text once in your index

Turn on removeRepeatedBlocks and paragraphs that already appeared on an earlier page of the same site are dropped: newsletter boxes, "related articles" lists, shared disclaimers and other blocks the main-content extractor keeps. Without it, a 300-page blog can put the same call-to-action into your vector database 300 times, and it then turns up in every search. Headings and code are always kept; each page reports repeatedBlocksRemoved.

Output (one item per page)

url, requestedUrl, title, markdown, characters, words, chunks[] (if chunkSize > 0), redactions, httpStatus, crawledAt, plus syncStatus in sync mode. Totals in SUMMARY; sync results in SYNC_REPORT.

Limits

• It reads the HTML the server sends (no browser), so text, links or markup added later by JavaScript are not seen.
• PDFs and other files are skipped (use our Document to Markdown OCR actor for those).
• Crawling stays inside the limits you set (maxDepth, path, include/exclude), so pages outside them are not fetched.

Responsible crawling

We follow each site's robots.txt for every page and sitemap (disallowed = skipped, free), send at most about one request per second per site (slower if the site asks for a crawl-delay), use an honest User-Agent (SiteToMarkdown/1.0), and never log in or bypass bot protection. Crawl sites you own or are allowed to reuse, and respect copyright when you use the content.

Pricing

Event	Price
Page converted to Markdown	$0.0018

A 500-page documentation site costs at most $0.90.

Maintained by DigiVibeTech. Questions or bugs: use the Issues tab; we reply within 2–5 business days.