Website to Markdown Crawler for RAG & llms.txt — Docsieve

Docsieve: Website to Markdown Crawler for RAG, LLMs & llms.txt

Turn any website or docs site into clean Markdown, RAG-ready chunks and llms.txt, and keep it fresh

Point Docsieve at a website or documentation site. It crawls it like a search engine (sitemap + internal links, robots.txt respected), strips menus, headers, footers, cookie banners and scripts, and converts each page's main content to clean Markdown with code fences (language kept), tables and absolute links. Every page comes with heading-aware chunks and token estimates, and the run also writes llms.txt and llms-full.txt for the site. On a schedule, monitor mode returns only new and changed pages, so your vector index stays current for pennies. $1 per 1,000 pages, no start fee.

▶ Try it now with a prefilled input · $0.001 per page (a 500-page docs site = $0.50) · llms.txt files free · Apify's free plan includes $5 of credit a month

Built for: developers building RAG chatbots and AI assistants on their own docs · AI agents that need a site's content as text · teams publishing llms.txt for AI search (GEO) · knowledge-base and support-bot builders · researchers collecting web text

Real output (real run on Apify, 10 Nov 2026: Apify Academy, "includePaths": ["/academy*"], 10 pages in 5.2 s, platform cost $0.0001; four rows shown):

Page	Words	Tokens (est.)	Chunks
/academy	254	489	1
/academy/advanced-web-scraping/crawling/sitemaps-vs-search	1,000	1,581	4
/academy/ai/ai-agents	1,321	2,540	6
/academy/anti-scraping	2,551	4,271	11

Start of the first page's Markdown:

# Apify Academy

Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.

## Beginner courses

#### Web scraping with AI

Learn how to use AI to extract information from websites in this practical course, starting from the absolute basics.

<https://docs.apify.com/academy/scraping-with-ai>

{"startUrls": ["https://docs.apify.com/academy"], "includePaths": ["/academy*"], "maxPages": 10}

Ready-made inputs (one click)

Job	Input	Result
Docs → RAG knowledge base	{"startUrls": ["https://docs.yourproduct.com"], "maxPages": 2000, "outputMode": "chunks"}	One row per chunk, ready to embed.
Generate llms.txt	{"startUrls": ["https://yoursite.com"], "maxPages": 300, "chunking": false}	LLMS-TXT and LLMS-FULL-TXT in the run's storage.
Just the blog	{"startUrls": ["https://site.com/blog"], "includePaths": ["/blog/*"], "excludePaths": ["/blog/tag/*"]}	Every post as Markdown.
Keep a chatbot fresh	{"startUrls": ["https://docs.yourproduct.com"], "monitor": true, "outputMode": "chunks"}	On a schedule: only new and changed pages; removed pages listed.
Several sites at once	{"startUrls": ["https://a.com", "https://b.com"], "maxPages": 100}	A summary and llms.txt per site.

Output

Pages mode (default): one row per page with url, title, description, lang, canonical, depth, markdown, wordCount, tokens, contentHash, crawledAt, chunkCount and chunks[] (index, heading, headings, text, chars, tokens).

Chunks mode: one row per chunk with the page fields plus chunkId (stable: content hash + index), chunkIndex, chunkCount, heading (path such as Guide > Install > Linux), text and chunkTokens. Upsert by chunkId into Pinecone, Qdrant, Weaviate, pgvector, Chroma or Milvus; LangChain and LlamaIndex read Apify datasets directly.

Key-value store: LLMS-TXT (llms.txt: site name, summary and a link list grouped by section), LLMS-FULL-TXT (every page's Markdown with its source URL) and OUTPUT (per site: pages converted, total tokens, pages skipped by robots.txt / noindex / thin content, failures, and in monitor mode the changed and removed URLs).

Run it on a schedule (fresh RAG index)

1. Save your input with "monitor": true as a task.
2. Schedules → Create: daily or weekly (cron 0 3 * * *).
3. Add a webhook "when the run succeeds" to your ingestion job (or a Make/n8n flow) that upserts the new chunks and deletes chunks of removedSinceLastRun pages. Unchanged pages cost nothing.

Integrations

• LangChain (Python): ApifyDatasetLoader(dataset_id=..., dataset_mapping_function=lambda d: Document(page_content=d["text"], metadata={"source": d["url"], "heading": d["heading"]}))
• LlamaIndex: the Apify reader with the same mapping.
• API (curl):

  curl -X POST "https://api.apify.com/v2/acts/cybermax~website-to-markdown/run-sync-get-dataset-items?token=YOUR_TOKEN" \
    -H "Content-Type: application/json" -d '{"startUrls": ["https://docs.apify.com/academy"], "maxPages": 10}'
  

• Make / Zapier / n8n: the Apify app or node, then your vector database's node.
• AI agents (MCP): add cybermax/website-to-markdown to the Apify MCP server (https://mcp.apify.com?actors=cybermax/website-to-markdown) so an agent can read a whole site as Markdown.

Why Docsieve

	Docsieve	Typical website-to-Markdown Actor on the Store
Price	$1 per 1,000 pages, no start fee	Most-used pay-per-page rivals: $1.56 to $3 per 1,000 pages, some with a start fee; range on the Store $0.20 to $30 (Store prices on 10 Nov 2026)
Chunks	Heading-aware, size and overlap you choose, token estimates, chunk-per-row mode	Often whole pages only
llms.txt	llms.txt + llms-full.txt built every run, free	Separate tools
Re-runs	Monitor mode: only changed pages, removed pages listed	Full re-crawl every time
Clean text	Main content only; code language, tables and absolute links kept	Varies

FAQ

Does it render JavaScript? No: it reads the HTML the server sends, which keeps it fast and cheap. That covers most docs sites, blogs and marketing sites (Docusaurus, GitBook, MkDocs, Mintlify, WordPress, Next.js with server rendering). Pages that only exist after client-side rendering come back thin and are skipped (and not charged).

How are tokens counted? Estimated at 4 characters per token, close to GPT and Claude tokenizers for English text.

Is it legal? Docsieve reads public pages and obeys robots.txt; it skips pages marked noindex unless you ask for them. Convert sites you own or have the right to use, and check each site's terms for how you may use its content.

Why not Apify's Website Content Crawler? It is excellent and renders JavaScript, but you pay its compute for every page on every run. Docsieve is a fixed $0.001 per page, builds chunks and llms.txt for you, and on scheduled runs charges only for pages that changed.

Also from CyberMax

• Inkpluck: Article Extractor · single articles or RSS feeds to clean text and Markdown.
• Sitemoss: Sitemap URL Extractor · every URL a site lists.
• Auditmoss: SEO Audit Crawler · SEO issues and broken links across a site.

Found this useful? A quick rating helps others find it.

© 2026 CyberMax. All rights reserved.