Pricing

Pay per usage

RAG Pipeline Scraper — Website to Markdown & JSONL

Transform any website into clean Markdown and JSONL ready for RAG pipelines, vector databases (Pinecone, Weaviate, Chroma), and LLM training. Removes ads, navigation, and boilerplate automatically.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Niu Yuchiao

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Why use this Actor?

Building a RAG (Retrieval-Augmented Generation) system requires clean, chunked text from your source documents. This Actor does the heavy lifting:

Crawls the entire site (or a single page) starting from a seed URL
Cleans the HTML — removes scripts, ads, navbars, cookie banners, sidebars
Converts to Markdown using a battle-tested HTML→MD library (Turndown + GFM)
Chunks the text with a configurable word count and 10% overlap for better retrieval
Outputs both individual Markdown files (Key-Value store) and a combined output.jsonl (Dataset + Key-Value store)

Works with any public website: documentation sites, blogs, news portals, product pages, wikis.

Input

Field	Type	Default	Description
`startUrl`	string	(required)	The URL to start crawling from
`maxPages`	number	`50`	Maximum number of pages to scrape
`chunkSize`	number	`300`	Target word count per JSONL chunk
`outputMarkdown`	boolean	`true`	Save per-page Markdown to Key-Value store
`outputJsonl`	boolean	`true`	Save chunks to Dataset and `output.jsonl`
`includeMetadata`	boolean	`true`	Add `url`, `title`, `chunkIndex`, `scrapedAt` to each chunk
`sameDomainOnly`	boolean	`true`	Only follow links on the same domain as `startUrl`

Example input

{
  "startUrl": "https://docs.example.com",
  "maxPages": 100,
  "chunkSize": 400,
  "outputMarkdown": true,
  "outputJsonl": true,
  "includeMetadata": true,
  "sameDomainOnly": true
}

Output

Dataset (JSONL chunks)

Each row in the Dataset is one text chunk, ready for embedding:

{
  "text": "Retrieval-Augmented Generation (RAG) combines a retrieval system with a language model...",
  "metadata": {
    "url": "https://docs.example.com/intro",
    "title": "Introduction to RAG",
    "chunkIndex": 0,
    "scrapedAt": "2025-01-01T00:00:00.000Z"
  }
}

The Dataset also contains one page_summary row per page:

{
  "_type": "page_summary",
  "url": "https://docs.example.com/intro",
  "title": "Introduction to RAG",
  "wordCount": 1240,
  "chunkCount": 5
}

Key-Value store

Key	Content
`page-1-docs-example-com-intro`	Full Markdown for that page
`output.jsonl`	All chunks concatenated as JSONL (one JSON object per line)
`STATS`	`{ pagesScraped, totalChunks, startUrl }`

Integrations

Pinecone

import json, openai, pinecone

with open("output.jsonl") as f:
    chunks = [json.loads(line) for line in f]

pc = pinecone.Pinecone(api_key="YOUR_KEY")
index = pc.Index("my-index")

for i, chunk in enumerate(chunks):
    embedding = openai.embeddings.create(
        input=chunk["text"], model="text-embedding-3-small"
    ).data[0].embedding
    index.upsert([(str(i), embedding, chunk["metadata"])])

Weaviate

import weaviate, json

client = weaviate.connect_to_local()
collection = client.collections.get("Document")

with open("output.jsonl") as f:
    for line in f:
        chunk = json.loads(line)
        collection.data.insert({"text": chunk["text"], **chunk["metadata"]})

LangChain

from langchain_community.document_loaders import JSONLoader

loader = JSONLoader(
    file_path="output.jsonl",
    jq_schema=".",
    content_key="text",
    metadata_func=lambda r, _: r.get("metadata", {}),
    json_lines=True,
)
docs = loader.load()

How it works

Crawl — Uses CheerioCrawler (fast, no-JavaScript HTML scraper) to fetch pages concurrently
Clean — Strips <script>, <style>, <nav>, <header>, <footer>, <aside>, cookie banners, ad containers, sidebars
Extract — Prefers <article>, <main>, [role="main"] elements; falls back to <body>
Convert — Runs the cleaned HTML through Turndown (with GitHub Flavored Markdown plugin) to produce clean Markdown
Chunk — Splits by word count with 10% overlap so context is preserved across chunk boundaries
Store — Saves Markdown files and JSONL chunks to Apify storage for easy download via API

FAQ

Does it handle JavaScript-rendered pages? No — this Actor uses a fast HTML-only scraper. For JavaScript-heavy SPAs, pair it with Apify's apify/web-scraper to render pages first.

What languages does it support? Any language. The scraper is language-agnostic — it converts whatever HTML it receives.

Can I scrape a single page without following links? Yes — set maxPages: 1 or sameDomainOnly: false with a specific URL.

How large can the site be? The default cap is 50 pages. Raise maxPages up to any number — the Actor handles large sites gracefully with concurrent crawling.

Pricing

This Actor is billed per output chunk. A typical documentation site with 50 pages generates roughly 200–500 chunks.

Feedback & Issues

Found a bug or want a feature? Open an issue via the Issues tab in Apify Store, or contact the author directly through the Apify platform.

Fast Website to Markdown & RAG JSONL Crawler

orbiscribe/website-rag-dataset-builder

Paste a homepage or sitemap and get clean Markdown, metadata, JSONL chunks, and source URLs for RAG at a low per-page price.

Orbiscribe Labs

LLM Markdown Crawler

sleek_waveform/llm-markdown-crawler

Crawl any website and extract clean, boilerplate-free Markdown optimized for LLMs, RAG pipelines, and AI training datasets. Uses Mozilla Readability to strip navigation and ads, then converts to clean Markdown. No browser required — fast and cheap.

Daniel Dimitrov

AI Web Content Crawler - Markdown for LLMs

intelscrape/ai-web-content-crawler

Crawl any website and extract clean Markdown optimized for LLM training, RAG pipelines, and AI knowledge bases - removes boilerplate and outputs structured JSON with URL, title, markdown, and metadata.

IntelScrape

RAG Web Crawler: Clean Markdown + Token-Sized Chunks

commonelements/rag-ready-crawler

Turn any website into embeddings-ready chunks for RAG and vector databases. Structure-aware token-sized chunking, clean LLM-ready markdown, per-chunk citations and metadata, dedup, and junk filtering. Pay per result, no surprise compute bills.

Harry Schoeller

Docs-to-RAG Optimizer

vamsi-krishna/docs-to-rag-optimizer

Convert public developer documentation into clean Markdown, semantic RAG chunks, token counts, duplicate hashes, JSONL exports, and quality warnings for AI assistants.

Vamsi Krishna

Website to RAG Markdown Crawler

knotted_tussock/rag-markdown-crawler

Crawl any website or docs site and export clean Markdown plus JSONL-style chunks for RAG, LLM apps, and AI agents.

Ralph T

AI Training Data Scraper - LLM and RAG-Ready

george.the.developer/ai-training-data-scraper

Extract web content formatted for LLM fine-tuning and RAG pipelines. Output in OpenAI JSONL, Claude JSONL, Markdown, or raw text.

George Kioko

Website Content Crawler

parseforge/website-content-crawler

Crawl any website and pull clean Markdown content ready for AI! Follow links across a whole domain and extract page text, titles, headings, images, and metadata. Perfect for building RAG pipelines, training datasets, knowledge bases, and vector databases. Start crawling content in minutes!

ParseForge

Website & PDF to RAG JSONL Crawler

orbiscribe/linked-pdf-website-rag-crawler

Paste webpage and PDF URLs and get Markdown, JSONL chunks, PDF inventory, source warnings, and RAG-ready records.

Orbiscribe Labs

AI Dataset Converter - Website to Training Data

boztek-ltd/ai-dataset-converter

Crawl websites and convert content into AI-ready formats: RAG chunks, fine-tuning JSONL, Q&A pairs, clean Markdown. Token-aware chunking, quality scoring, deduplication. No external LLM API needed.

Boztek LTD

Website Content to Markdown for LLM Training

easyapi/website-content-to-markdown-for-llm-training

🚀 Transform web content into clean, LLM-ready Markdown! 📘 Scrape multiple pages, extract main content, and convert to Markdown format. Perfect for AI researchers, data scientists, and LLM developers. Fast, efficient, and customizable. Supercharge your AI training data today! 🌐📝🧠