Pricing

Pay per event + usage

RAG-Ready Website Crawler — Clean Content for LLMs & Vector DBs

Crawl websites and output clean, chunked content optimized for RAG pipelines, LLM training data, and vector databases. Built for AI knowledge bases and semantic search.

Pricing

Pay per event + usage

Rating

0.0

(0)

Developer

Wingman

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Actor: RAG-Ready Website Crawler

Crawl any website and output clean, chunked content optimized for Retrieval-Augmented Generation (RAG) pipelines, LLM training data, and vector databases.

Features

Clean content extraction: Removes navigation, ads, sidebars, footers
Semantic chunking: Smart split by headings and paragraphs
Markdown output: Clean markdown format ideal for LLM consumption
Metadata extraction: Title, description, headings, word count, publish date
Token counting: Track approximate token counts per chunk
Same-domain crawling: Respect site boundaries by default
Flexible crawling: Configurable page limits and depth

Use Cases

Build RAG knowledge bases from documentation sites
Collect training data for LLM fine-tuning
Create vector database entries from web content
Archive important web content in clean format
Build AI-ready datasets from blogs and documentation

Input

{
  "startUrls": ["https://docs.example.com"],
  "maxPages": 50,
  "chunkSize": 2000,
  "chunkOverlap": 200,
  "includeMetadata": true,
  "outputFormat": "markdown",
  "excludeSelectors": ["nav", "footer", ".sidebar", ".advertisement"]
}

Output

{
  "url": "https://docs.example.com/getting-started",
  "title": "Getting Started",
  "chunkIndex": 0,
  "content": "# Getting Started\n\nWelcome to the documentation...",
  "tokenCount": 450,
  "charCount": 1850,
  "headings": ["Getting Started", "Installation", "Quick Start"],
  "metadata": {
    "description": "Learn how to get started with Example",
    "author": "Example Team",
    "publishDate": "2026-01-15"
  }
}

Changelog

v1.0.0 (2026-07-09)

Initial release
Clean HTML-to-markdown content extraction
Semantic chunking with configurable size/overlap
Full metadata extraction
Token counting for LLM context estimation
Apify Dataset output with chunked entries

RAG Website Crawler - Markdown Chunks for LLMs & MCP

themineworks/rag-crawler

Crawl any website into clean, pre-chunked Markdown with per-chunk token counts for RAG pipelines, vector DBs (Pinecone, Qdrant) and LLM context. MCP-native for Claude & ChatGPT. SPA support via Playwright. Pay only for pages that crawl. A Firecrawl alternative.

The Mine Works

Website Content Crawler Pro

datascoutapi/website-content-crawler-pro

Crawl websites and extract clean, structured content in Markdown, JSON, or plain text for AI models, LLMs, vector DBs, or RAG pipelines. Fast, reliable, and stealthy, with bulk processing, advanced metadata extraction, and seamless integration with LangChain, LlamaIndex, and AI workflows.

halam

558

3.7

(3)

PDF to Markdown & JSON (RAG-Ready)

basisweb/pdf-to-markdown-rag

Convert PDFs to clean Markdown and structured JSON (text + tables) for RAG, LLMs, and vector DBs. Batch URLs, pay per page.

BasisWeb

Docs-to-RAG Crawler

automation-lab/docs-rag-crawler

Crawl documentation sites (ReadTheDocs, GitBook, Docusaurus, Mintlify) into RAG-ready Markdown/JSON chunks with stable chunk IDs, heading breadcrumbs, word counts, and token estimates.

Stas Persiianenko

AI-Powered Smart Web Scraper

cloud9_ai/ai-web-scraper

Intelligent content extraction from any website using Crawlee + AI. Auto-detects structure, adapts to layout changes, handles JavaScript rendering. No custom code needed. Extract articles, products, listings from 1000s of pages.

cloud9

Rag Content Chunker

labrat011/rag-content-chunker

Turn raw text, Markdown, or Apify datasets into token-perfect RAG chunks with deterministic IDs, source metadata, and a billing-ready summary—ready for embeddings or vector DBs without extra glue code.

mick_

RAG-Ready Markdown Converter & Chunker

foxpink/apify-rag-markdown-chunker

Convert raw HTML/text into clean Markdown and split into ready-to-ingest chunks for RAG pipelines, Vector DBs, and LLM fine-tuning workflows.

Nguyễn Anh Duy

4.7

(3)

RAG Web Crawler: Clean Markdown + Token-Sized Chunks

commonelements/rag-ready-crawler

Turn any website into embeddings-ready chunks for RAG and vector databases. Structure-aware token-sized chunking, clean LLM-ready markdown, per-chunk citations and metadata, dedup, and junk filtering. Pay per result, no surprise compute bills.

Harry Schoeller

Dark Web Scraper – Onion Search & Page Extractor

lofomachines/tor-scraper

Search the dark web by keyword and convert any .onion page into clean Markdown, JSON, HTML or text with rich metadata.

Lofomachines

HTML to Markdown Converter - Bulk Web Content to MD

santamaria-automations/html-to-markdown

Extract main article content from any website and convert to clean Markdown including headings, links, images, tables, and code blocks. Perfect for LLM training, AI pipelines, and documentation. Export data, run via API, schedule and monitor runs, or integrate with other tools.