🤖 RAG Website Crawler: Markdown Chunks for LLMs & MCP

Overview

RAG Website Crawler turns any website into a clean, pre-chunked Markdown corpus with per-chunk token counts, ready to drop straight into a RAG pipeline, vector database (Pinecone, Qdrant, Weaviate), or LLM context window. Point it at one or more URLs, choose whether to render JavaScript SPAs with Playwright, and it crawls, strips boilerplate, converts to Markdown, and splits by heading. Output is structured JSON you can index immediately.

Built for AI teams, chatbot builders, and any workflow that needs website content in LLM-native shape without writing HTML cleanup or chunking code.

✅ No login | ✅ SPA support via Playwright | ✅ Token-counted chunks | ✅ MCP-ready for AI agents

Features

Crawl one or many start URLs with configurable page cap. Optional JavaScript rendering via Playwright for SPAs and React sites. Boilerplate stripping to remove nav, footer, and cookie-banner noise. Heading-aware Markdown chunking with per-chunk token counts. Same output whether you consume it via CSV, JSON, Excel, API, or MCP tool call.

How it works

Most RAG pipelines fail at the source: raw HTML full of nav bars and pop-ups, unpredictable chunk sizes, and no rendering for JavaScript apps. RAG Website Crawler handles all of it. It fetches static HTML fast for the common case, upgrades to Playwright rendering when you flip renderJs, and then converts each page to clean Markdown. From there it splits by heading and computes accurate token counts per chunk, so you can plan embedding costs and context budgets before indexing.

The output is one record per crawled page. Each record carries the canonical URL, full-page Markdown, and an array of pre-cut chunks with individual token counts. You get a Firecrawl-style feed at a fraction of the cost, with the same clean shape.

🧾 Input configuration

{
  "startUrls": [{ "url": "https://docs.stripe.com" }],
  "maxPages": 500,
  "renderJs": false,
  "waitStrategy": "domcontentloaded",
  "outputFormat": "markdown",
  "maxTokensPerChunk": 500,
  "stripBoilerplate": true,
  "excludeUrlPatterns": ["/blog/", "/changelog/"],
  "customCss": "main.content"
}

📤 Output format

{
  "url": "https://en.wikipedia.org/wiki/Web_scraping",
  "canonical": "https://en.wikipedia.org/wiki/Web_scraping",
  "title": "Web scraping - Wikipedia",
  "description": "",
  "language": "en",
  "word_count": 4445,
  "token_count": 6001,
  "crawled_at": "2026-07-15T04:20:49.141Z",
  "status": "success",
  "chunks": [
    {
      "heading_path": [],
      "text": "Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites. Web scraping software may directly access the World Wide Web using the Hypertext Transfer Protocol or a web browser...",
      "token_count": 185,
      "chunk_index": 0
    },
    {
      "heading_path": ["Human copy-and-paste"],
      "text": "The simplest form of web scraping is manual copying and pasting of data from a web page into a text file or spreadsheet...",
      "token_count": 134,
      "chunk_index": 5
    }
  ],
  "markdown": "Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites..."
}

This is a genuine record from a live crawl of the real Wikipedia page above. Note the actual chunk shape is {heading_path, text, token_count, chunk_index}, where heading_path is an array of parent headings (empty for lead-section text), not a single heading string. status is a string ("success" or "failed"), not an HTTP status code.

Every page record contains these fields:

Field	Description
🌐 url	Fetched URL
🔗 canonical	Canonical URL from <link rel="canonical"> if present
🏷️ title	Page title
📝 description	Meta description
🈁 language	Detected page language code
🔢 word_count	Word count of the cleaned page
🎯 token_count	Token count of the full page (LLM planning)
🕒 crawled_at	ISO timestamp of crawl
✅ status	success or failed
📚 chunks	Array of {heading_path, text, token_count, chunk_index} chunks
📄 markdown	Full cleaned Markdown body

💼 Common use cases

RAG and vector-database ingest Feed Pinecone, Qdrant, or Weaviate with heading-aware, token-sized chunks with zero glue code. Index a docs site nightly to keep an AI copilot current.

AI chatbot knowledge bases Build a support bot on a product's docs, help center, or public wiki. Wire a Claude or ChatGPT MCP flow directly to your knowledge base.

LLM context and prompt engineering Preview token counts before you spend on embeddings or context. Assemble a curated context bundle for a long-context model like Claude Opus.

Content and market intelligence Crawl competitor sites into structured Markdown for downstream analysis. Build a searchable archive of a domain for research or reporting.

🚀 Getting started

1. Open the actor and set startUrls to the pages or roots you want crawled.
2. Set maxPages to cap the crawl budget.
3. Turn on renderJs if the site is a React or Vue SPA that needs a browser.
4. Choose outputFormat (markdown is default), tune maxTokensPerChunk, and set excludeUrlPatterns for noise.
5. Click Start, then download the dataset as JSON, CSV, or Excel, or pull it via API or MCP.

FAQ

How is this different from Firecrawl? Same output shape (clean Markdown, token counts, chunk splits), fully pay-per-result on Apify infrastructure, and available as a native MCP tool. No monthly minimum.

Does it support JavaScript-heavy sites? Yes. Flip renderJs on and the actor uses Playwright to render the DOM before extracting Markdown.

How does the chunking work? It splits by top-level headings first, then packs sections into chunks up to maxTokensPerChunk tokens using an accurate tokenizer. Each chunk keeps its parent heading so context is preserved for retrieval.

How much does it cost? $2 per 1,000 pages successfully crawled. Empty or failed runs cost nothing. First 25 pages on every account are free for life.

Can I use it inside an AI agent? Yes. It is exposed as an MCP tool. See below.

Use in Claude, ChatGPT & any MCP agent

https://mcp.apify.com/?tools=themineworks/rag-crawler

Or call it programmatically with the Apify client:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });

const run = await client.actor('themineworks/rag-crawler').call({
  startUrls: [{ url: 'https://docs.stripe.com' }],
  maxPages: 500,
  renderJs: false,
  maxTokensPerChunk: 500,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

🛠️ Complete your knowledge-base pipeline

Pair this actor with the rest of the themineworks AI and data suite:

• GitHub Skill & Memory Discovery: pull SKILL.md, CLAUDE.md, and AGENTS.md content for AI-agent knowledge bases.
• YouTube Transcript Scraper: add spoken-word transcripts to a text corpus.
• Federal Register Scraper: index authoritative government text for a compliance copilot.

Typical flow: rag-crawler handles the docs, youtube-transcript-scraper covers video, federal-register-scraper adds regulatory context, all in one dataset ready for embedding.

Questions or need a custom field set? Reach out through the Apify profile.