Universal RAG Web Scraper

Pricing

from $0.01 / 1,000 results

Universal RAG Web Scraper

Turn any website into clean, LLM-ready Markdown. Automatically strips ads, navigation, and noise using Mozilla Readability. Perfect for feeding data to ChatGPT, Claude, or Vector Databases (RAG).

Pricing

from $0.01 / 1,000 results

Rating

0.0

(0)

Developer

Prince Raj

Maintained by Community

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

🕷️ Universal RAG Web Scraper

Turn any website into LLM-ready Markdown.

This Actor is designed specifically for AI engineers and developers building RAG (Retrieval-Augmented Generation) pipelines. It scrapes websites and converts them into clean, structured Markdown, stripping away noise like navigation bars, ads, and footers.

🚀 Why use this Actor?

LLM-Optimized Output: Uses Mozilla's Readability engine + Turndown to produce noise-free Markdown.
Metadata Rich: Extracts title, byline, published time, and site name for better vector embedding.
Smart Chunking Ready: The output format is perfect for semantic chunking strategies.
Deep & Focused: Supports glob patterns to scrape complete documentation sections (e.g., **/docs/**).

🛠️ Usage

Inputs

Field	Type	Description	Default
`startUrls`	Array	List of URLs to start crawling.	(Required)
`maxDepth`	Integer	How deep to crawl. `0` = single page, `1` = links on page.	`1`
`includeGlobs`	Array	Only visit URLs matching these patterns (e.g., `/blog/`).	`[]`
`excludeGlobs`	Array	Skip URLs matching these patterns.	`[]`
`cleanMarkdown`	Boolean	Remove images and links for text-only output.	`false`

Output Example

{
  "url": "https://example.com/blog/article",
  "title": "How to Build a Scraper",
  "description": "A comprehensive guide to scraping.",
  "markdown": "# How to Build a Scraper\n\nScraping is fun...",
  "metadata": {
    "siteName": "Example Blog",
    "publishedTime": "2023-10-01"
  },
  "crawledAt": "2023-10-25T10:00:00.000Z"
}

📦 Integration

Easily integrate with LangChain, Haystack, or your custom Python/JS scripts using the Apify API.

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("antigravity/rag-web-scraper").call(run_input={
    "startUrls": [{"url": "https://react.dev/reference/react"}],
    "maxDepth": 2,
    "includeGlobs": ["**/reference/**"]
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["markdown"])

🏆 Goals

Built to solve the "Context Retrieval" problem for modern AI applications.

Universal Markdown Scraper for LLMs

botflowtech/universal-markdown-scraper-for-llms

Universal Markdown Scraper for LLMs

BotFlowTech

Website Content Crawler Pro

datascoutapi/website-content-crawler-pro

Crawl websites and extract clean, structured content in Markdown, JSON, or plain text for AI models, LLMs, vector DBs, or RAG pipelines. Fast, reliable, and stealthy, with bulk processing, advanced metadata extraction, and seamless integration with LangChain, LlamaIndex, and AI workflows.

halam

218

5.0

(1)

Universal Web Extractor V8

motivational_nickel/my-actor

Flexible web extractor using Python + Playwright or HTTP. Supports CSS-based field extraction, HTML snapshots, screenshots, metadata, monitoring mode, and link-following. Ideal for scraping product pages, listings, news articles, tech profiles, or universal structured data from any website.

Leoncio Jr Coronado

5.0

(1)

Hybrid Vision Spider | AI-Powered Universal Web Scraper

tuguidragos/hybrid-vision-spider-ai-powered-universal-web-scraper

AI-driven hybrid web scraper that merges Playwright and Vision intelligence to extract structured data from any dynamic site. Schema-aware, proxy-ready, budget-safe, and fully compatible with Apify datasets.

Țugui Dragoș

5.0

(2)

Scrape GPT - Universal AI Web Scraper Agent

paradox-analytics/scrape-gpt---universal-ai-web-scraper-agent

AI-powered universal web scraper that works on ANY website without configuration. Extract data from e-commerce, news sites, social media, and more using intelligent LLM-based field mapping. Features JSON-first extraction, automatic pagination, anti-bot bypass, and cost-effective caching.

Paradox Analytics

5.0

(1)

RAG Spider - Web to Markdown Crawler for AI Training Data

lenient_grove/RAG-Spider

Enterprise-grade web crawler that converts messy websites into clean, chunked Markdown for AI systems. Uses Mozilla Readability for 95% cleaner extraction than competitors. Outputs RAG-ready data with metadata and token estimates. Perfect for building knowledge bases and training AI chatbots.

Tejas Rawool

Web Text Extractor

rl1987/web-text-extractor

R.L.

Universal Article Scraper

universal_scraping/universal-article-scraper

Universal article scraper for news websites, blogs, etc. It can scrape articles from multiple websites simultaneously, including metadata such as title, content, publication date, image, and author.

Michael Novak

5.0

(2)

GraphQL Extractor

jupri/graphql-extractor

💫 Universal GraphQL Scraper

cat

Linkedin Enhanced Mcp Server

shubham_palriwala/linkedin-enhanced-mcp-server

A comprehensive MCP server providing 5 LinkedIn scraping tools and research prompts powered by Apify actors. Extract profiles, analyze posts, search jobs, and conduct professional research with built-in intelligence workflows.

Shubham Palriwala