Pricing

from $2.00 / 1,000 results

Try for free

Go to Apify Store

Website Crawler - Markdown Scraper for AI & RAG

Try for free

Crawl public websites, docs, blogs, and help centers into clean Markdown. Extract content with source URLs and page titles, or split it into chunks for RAG, chatbots, and AI agents. Export JSON or CSV. HTTP crawler for server-rendered pages; no AI API key needed.

Pricing

from $2.00 / 1,000 results

Rating

0.0

(0)

Developer

Group Oject

Actor stats

Bookmarked

Total users

Monthly active users

19 hours ago

Last modified

Categories

Automation

Agents

You can access the Website Crawler - Markdown Scraper for AI & RAG programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1import { ApifyClient } from 'apify-client';
2
3// Initialize the ApifyClient with your Apify API token
4// Replace the '<YOUR_API_TOKEN>' with your token
5const client = new ApifyClient({
6    token: '<YOUR_API_TOKEN>',
7});
8
9// Prepare Actor input
10const input = {
11    "startUrls": [
12        {
13            "url": "https://docs.apify.com/"
14        }
15    ],
16    "proxyConfiguration": {
17        "useApifyProxy": true
18    }
19};
20
21// Run the Actor and wait for it to finish
22const run = await client.actor("groupoject/ai-rag-web-crawler").call(input);
23
24// Fetch and print Actor results from the run's dataset (if any)
25console.log('Results from dataset');
26console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
27const { items } = await client.dataset(run.defaultDatasetId).listItems();
28items.forEach((item) => {
29    console.dir(item);
30});
31
32// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

Website Crawler - Markdown Scraper for AI & RAG API in JavaScript

The Apify API client for JavaScript is the official library that allows you to use Website Crawler - Markdown Scraper for AI & RAG API in JavaScript or TypeScript, providing convenience functions and automatic retries on errors.

Install the apify-client

$npm install apify-client

Other API clients include:

Website Crawler - Markdown Scraper for AI & RAG API in Python

Website Crawler - Markdown Scraper for AI & RAG API through CLI

Website Crawler - Markdown Scraper for AI & RAG OpenAPI definition

Website Crawler - Markdown Scraper for AI & RAG API

Web-to-Markdown Generator for AI & RAG Pipelines

profitstack/web-to-markdown-generator-for-ai-rag-pipelines

Convert any website into clean, heading-based chunking, LLM-ready Markdown for RAG and AI agents.

Manas Mantri

RAG Web Browser — Web Content for AI & LLMs

nexgendata/rag-web-browser

Web browser for RAG pipelines and AI agents. Search Google, scrape top results, return clean Markdown. Feed your LLM with real-time web data. Works with Claude, GPT, LangChain, CrewAI. No API key needed.

NexGenData

Rag Web Browser

opalescent_quintet/rag-web-browser

A specialized crawler designed exclusively to feed LLMs. It visits a website and extracts core content into clean, token-optimized Markdown, stripping all "junk" (navs, footers, ads, cookie banners).

Aryan

YouTube Subtitles & AI LLM Context

unrivaled_fortress/youtube-transcript-summary-ai

Extract full transcripts, auto-captions, and timestamps from any YouTube video in seconds. Formatted for ChatGPT, Claude, RAG, and AI summarizers.

David A

Website Content Extractor - Markdown for LLM, RAG, Vector DB

snow_leo_data/website-to-markdown

Readable text from 73.1% of 52 live pages over plain HTTP, including 10 of 10 documentation sites. Website to LLM text and a website to markdown converter in one run: clean Markdown, chunks with overlap for a vector database, 46 flat fields, and what changed since the last run.

Snow Leo Data

Website & PDF to RAG JSONL Crawler

orbiscribe/linked-pdf-website-rag-crawler

Paste webpage and PDF URLs and get Markdown, JSONL chunks, PDF inventory, source warnings, and RAG-ready records.

Orbiscribe Labs

RAG Markdown Scraper

tillzero/rag-markdown-scraper

Turn any website into a clean Vector Database pipeline. Bypasses bot protections to extract pure GitHub-Flavored Markdown. Features built-in AI Vision OCR for images, Delta Crawling to save costs, Custom Regex, and direct Pinecone upserts for RAG agents.

K S

RAG-Ready Web Scraper & Smart Chunker for AI Knowledge Bases

adinfosys-labs/rag-ready-web-scraper-smart-chunker-for-ai-knowledge-bases

RAG-ready web scraper that collects, cleans, deduplicates, filters, and chunks web content into structured datasets for AI pipelines. Generates high-quality knowledge-base data optimized for LLMs, embeddings, and vector databases

Artashes Arakelyan

🧩 RAG Dataset Builder - Crawl to Embedding-Ready Chunks

that_red_bird/rag-dataset-builder

🧩 Crawl a site and get back a FINISHED, embedding-ready RAG dataset — not just clean text. ✅ Semantic chunking on heading boundaries, never mid-code-block or mid-table. ✅ Token estimator, configurable overlap, near-duplicate removal, full provenance per chunk.

mohamed alaya

RAG Web Browser Lite — Cheap Markdown for AI Agents

subimpact/rag-web-browser-lite

Web search and fetch tool for AI agents and RAG pipelines. Queries Google Search, scrapes the top N pages over raw HTTP, and returns clean Markdown. Fixed $0.002 per page — no surprise compute bills.