Website to Markdown & RAG Crawler: Clean Text for LLM
Pricing
from $2.00 / 1,000 extracted rag pages
Website to Markdown & RAG Crawler: Clean Text for LLM
Convert entire websites into clean Markdown and semantic token chunks for LLMs, RAG vector pipelines, Claude, and OpenAI embeddings.
Pricing
from $2.00 / 1,000 extracted rag pages
Rating
0.0
(0)
Developer
Lakshay Papneja
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
18 hours ago
Last modified
Categories
Share
AI Website to Markdown & RAG Chunk Extractor
Convert any website, documentation portal, blog, or knowledge base into clean Markdown, token-measured semantic RAG chunks, and structured JSON-LD metadata ready for LangChain, LlamaIndex, Pinecone, Qdrant, Weaviate, and MCP AI Agents.
Why Use This Actor?
- 10x Faster & Cheaper: Uses lightweight HTTP + custom HTML-to-Markdown parsing instead of heavy headless browsers wherever possible.
- Built-in Semantic Chunking: Automatically splits pages by heading hierarchy (
#,##,###) and configurable token windows (chunkSizeTokens+chunkOverlapTokens). - Complete Metadata: Extracts OpenGraph tags, author, publish dates, and embedded
application/ld+jsonschemas. - MCP Ready: Designed to work seamlessly with
mcp.apify.comso AI assistants can fetch and read live web documentation in a single tool call.
📖 How to Use (Step-by-Step)
- Set Start URLs: Enter the root documentation or website URL (e.g.
https://docs.apify.com/platform/actors). - Configure Chunking: Set
chunkSizeTokens(default: 500) andchunkOverlapTokens(default: 50) to match your LLM embedding model. - Set Noise Selectors: Leave
removeSelectorsas default to strip headers, footers, cookie banners, navigation, and ads. - Select Max Pages: Choose how many pages to crawl (supports up to 1,000 pages).
- Run & Embed: Download clean Markdown and token-measured chunk vectors for Pinecone, Weaviate, Chroma, or LangChain.
💰 Transparent Pay-Per-Event Pricing
- Actor Start Fee: $0.00005
- Per Extracted Page: $0.002 ($2.00 per 1,000 clean markdown pages)
- Free Monthly Tier: Any Apify user with their $5/month free credit can crawl up to 2,500 documentation pages completely free.
🏆 Why Use This Actor vs Firecrawl or Crawl4AI?
| Feature | AI Web to Markdown RAG | Heavy Browser Crawlers |
|---|---|---|
| Compute Footprint | ⚡ Ultra-light (128MB–256MB RAM) | 💸 Heavy (4GB–8GB RAM) |
| Semantic Chunking | ✅ Built-in with token counter | ❌ Raw unchunked markdown |
| Noise Filtering | ✅ Strips ads, navs, footers, cookie banners | ⚠️ Boilerplate leaks |
| JSON-LD Schema Extraction | ✅ Preserves structured metadata | ❌ Stripped away |
| MCP AI Server Integration | ✅ Direct mcp.apify.com tool support | ❌ Requires custom API proxy |
❓ Frequently Asked Questions (FAQ)
Which LLM embedding models is this chunking optimized for?
The 500-token chunking with 50-token overlap is optimized for OpenAI text-embedding-3-small, text-embedding-3-large, Cohere embed-english-v3.0, and open-source Hugging Face BGE embeddings.
Does it follow internal links?
Yes, when followInternalLinks is enabled, it discovers same-origin links and crawls documentation hierarchies up to maxPages.
How are code blocks handled?
Code blocks (<pre><code>) are preserved intact with language identifiers, ensuring code examples remain uncorrupted for coding LLMs (Claude 3.5 Sonnet, GPT-4o, Cursor).
