LLM-Ready Web Scraper

Pricing

$2.50/month + usage

LLM-Ready Web Scraper

Convert web pages to clean, LLM-friendly text. Perfect for RAG pipelines, AI chatbot training, and fine-tuning datasets. Removes ads,menus, and clutter automatically.

Pricing

$2.50/month + usage

Rating

0.0

(0)

Developer

batuhan senavci

Maintained by Community

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

Use Cases

RAG Pipelines: Get chunked content ready for vector databases
Fine-tuning Datasets: Export as JSONL for LLM training
Knowledge Bases: Build AI chatbot training data
Content Extraction: Clean text without ads, menus, or clutter

Features

Automatic content extraction (removes ads, navigation, footers)
Multiple output formats: Markdown, JSON, JSONL
Optional chunking with overlap for RAG
Batch URL processing
Metadata extraction (title, description, domain)

Output Formats

Markdown

---
title: "Page Title"
url: https://example.com/page
domain: example.com
scraped_at: 2024-01-15T10:30:00Z
---

Clean page content here...

JSON

{
  "url": "https://example.com",
  "success": true,
  "content": "Clean text content...",
  "metadata": {
    "title": "Page Title",
    "description": "Meta description"
  },
  "word_count": 1500
}

JSONL (Fine-tuning)

{
  "prompt": "Content from Page Title:",
  "completion": "Clean text content..."
}

With Chunks (RAG-ready)

{
  "chunks": [
    {"chunk_id": 0, "text": "First chunk...", "word_count": 500},
    {"chunk_id": 1, "text": "Second chunk...", "word_count": 500}
  ],
  "chunk_count": 5
}

Input Parameters

Parameter	Type	Default	Description
url	string	-	Single URL to scrape
urls	array	-	Multiple URLs for batch processing
outputFormat	string	markdown	Output format: markdown, json, jsonl
includeChunks	boolean	false	Split into RAG-ready chunks
chunkSize	integer	500	Words per chunk
chunkOverlap	integer	50	Overlap between chunks
maxConcurrency	integer	5	Parallel scraping limit

Example Input

{
  "urls": [
    "https://docs.python.org/3/tutorial/",
    "https://docs.python.org/3/library/"
  ],
  "outputFormat": "json",
  "includeChunks": true,
  "chunkSize": 500
}

Pricing

Pay only for what you use. Typical cost: $0.01-0.05 per URL depending on page size.

Tech Docs to LLM-Ready Markdown

hedelka/tech-docs-scraper

Scrapes technical documentation sites (Docusaurus, GitBook, MkDocs, ReadTheDocs) and converts them to clean, structured Markdown for RAG pipelines, LLM training, and AI assistants. Automatically detects documentation framework and removes navigation elements.

Dmitry Goncharov

Clean Web Scraper - Markdown for AI 🔥 Firecrawl API

clearpath/web-to-markdown

Convert any website to clean, LLM-optimized markdown using Firecrawl. Perfect for RAG pipelines, AI training data, and knowledge bases. No login required, 25% cheaper than Firecrawl direct. Batch process hundreds of URLs. Supports PDF/DOCX. Pay only $0.004 per page - no monthly fees.

ClearPath

Universal RAG Web Scraper

express_kingfisher/rag-web-scraper

Turn any website into clean, LLM-ready Markdown. Automatically strips ads, navigation, and noise using Mozilla Readability. Perfect for feeding data to ChatGPT, Claude, or Vector Databases (RAG).

Prince Raj

Scrape Failed Killer: Deep URL Content Crawler

6sigmag/scrape-failed-killer-deep-url-content-crawler

Scrape Failed Killer! A high-performance web scraper that rapidly extracts and analyzes content from multiple URLs simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David Deng

Website To LLM Knowledge Pack

attainable_iota/website-to-llm-knowledge-pack

Crawl any website and turn it into an LLM-ready knowledge pack. This Actor extracts clean main text + metadata, follows links with depth/URL filters, and outputs per-page dataset items plus knowledge.jsonl, knowledge.md, and manifest.json for RAG/embeddings pipelines.

M Junaid Shaukat

Quick Website Content Scraper ( Extract Text for RAG & LLMs )

automateitplease/ai-web-content-scraper-extract-text-for-rag-llms

Extract clean text from any website for AI/LLM applications. Supports both static and JavaScript-rendered sites (React, Vue, Angular). Perfect for RAG systems, chatbot training, and content analysis.

AutomateItPlease Workflow And Automaton Ops

Rag Knowledge Graph Builder

cspnair/rag-knowledge-graph-builder

Transform websites into RAG-ready datasets. Crawls pages, chunks content into semantic segments (500-1000 tokens), and generates hypothetical questions for each chunk. No API key needed with native mode. Output: pre-indexed JSON optimized for AI retrieval with 3x better accuracy than raw text.

csp

5.0

(8)

Universal Markdown Scraper for LLMs

botflowtech/universal-markdown-scraper-for-llms

Universal Markdown Scraper for LLMs

BotFlowTech

Web Scraper 🚀

datascoutapi/web-scraper

Web Scraper Pro extracts clean structured data for LLMs/RAG. Browser-based, 10x faster with anti-detection bypassing Cloudflare/CAPTCHA & proxy rotation. Bulk/recursive crawl 50k URLs at 500 pages/min. JSON/CSV/API, free tier.

halam

Hyper Reader

ai_solutionist/hyper-reader

High-fidelity web extraction for AI agents. Clean Markdown optimized for Claude, GPT-4 & Gemini. 3-level stealth, Vision screenshots, Deep Read link following. Standby Mode for 1-second responses.