Pricing

from $3.00 / 1,000 page processeds

Website & Docs to Markdown + RAG Chunks

Turn websites & docs into clean Markdown plus token-bounded, embeddings-ready RAG chunks (heading lineage + sha256) ready for Pinecone, Weaviate, Qdrant or pgvector. Optional no-hallucination field extraction and AEO mode (FAQ, answer-first, llms.txt, citations). Robots honored; ownership required.

Pricing

from $3.00 / 1,000 page processeds

Rating

0.0

(0)

Developer

Adam

Actor stats

Bookmarked

Total users

Monthly active users

15 hours ago

Last modified

Categories

Developer tools

SEO tools

You can access the Website & Docs to Markdown + RAG Chunks programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1import { ApifyClient } from 'apify-client';
2
3// Initialize the ApifyClient with your Apify API token
4// Replace the '<YOUR_API_TOKEN>' with your token
5const client = new ApifyClient({
6    token: '<YOUR_API_TOKEN>',
7});
8
9// Prepare Actor input
10const input = {
11    "source": {
12        "mode": "url_list",
13        "urls": [
14            "https://docs.example.com/guide"
15        ],
16        "maxPages": 1000
17    },
18    "chunking": {
19        "strategy": "semantic",
20        "maxTokens": 512,
21        "overlapTokens": 64
22    },
23    "extractSchema": {
24        "fields": [
25            {
26                "name": "product_name",
27                "type": "string",
28                "instruction": "The product name stated on the page; null if absent."
29            }
30        ]
31    },
32    "aeo": {
33        "assets": [
34            "faq_pair",
35            "answer_block",
36            "llms_txt"
37        ]
38    }
39};
40
41// Run the Actor and wait for it to finish
42const run = await client.actor("awesome_highboy/aeo-rag-ready-content-structurer").call(input);
43
44// Fetch and print Actor results from the run's dataset (if any)
45console.log('Results from dataset');
46console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
47const { items } = await client.dataset(run.defaultDatasetId).listItems();
48items.forEach((item) => {
49    console.dir(item);
50});
51
52// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

Website to Markdown + RAG Chunks for Embeddings & LLMs API in JavaScript

The Apify API client for JavaScript is the official library that allows you to use Website & Docs to Markdown + RAG Chunks API in JavaScript or TypeScript, providing convenience functions and automatic retries on errors.

Install the apify-client

$npm install apify-client

Other API clients include:

Website & Docs to Markdown + RAG Chunks API in Python

Website & Docs to Markdown + RAG Chunks API through CLI

Website & Docs to Markdown + RAG Chunks OpenAPI definition

Website & Docs to Markdown + RAG Chunks API

Clean Web Scraper - Markdown for AI via Firecrawl

clearpath/web-to-markdown

Convert any website to clean, LLM-optimized markdown using Firecrawl. Perfect for RAG pipelines, AI training data, and knowledge bases. No login required, 25% cheaper than Firecrawl direct. Batch process hundreds of URLs. Supports PDF/DOCX. Pay only $0.004 per page - no monthly fees.

ClearPath

Website To Markdown

smart_api/website-to-markdown

Convert any webpage into clean, LLM-ready Markdown in seconds — perfect for AI training data, RAG pipelines, and content archiving.

SmartApi

5.0

(1)

Website Content Crawler for LLM's

salesblaster-ai/website-content-crawler

Extract contact information + turn any website into clean, structured content ready for LLM's (e.g. AI lead magnets, RAG pipelines, and outbound personalization). Most web scrapers dump raw HTML or unstructured text. This crawler is purpose-built for LLM's, and optimized for lead generation.

SalesBlaster AI

RAG-Ready Web Scraper & Smart Chunker for AI Knowledge Bases

adinfosys-labs/rag-ready-web-scraper-smart-chunker-for-ai-knowledge-bases

RAG-ready web scraper that collects, cleans, deduplicates, filters, and chunks web content into structured datasets for AI pipelines. Generates high-quality knowledge-base data optimized for LLMs, embeddings, and vector databases

Artashes Arakelyan

Docs Markdown Rag Ready Crawler

devwithbobby/docs-markdown-rag-ready-crawler

Turn any documentation site or website into clean, structured markdown—ready for RAG, embeddings, and AI agents.

Dev with Bobby

Website Content Scraper

qaseemiqbal/website-content-scraper

Extract clean Markdown, plain text, linked files, and RAG-ready chunks from websites, documentation, help centers, knowledge bases, and authenticated portals. Preserve structure, metadata, URLs, and crawl context for AI search, training, and retrieval workflows.

Muhammad Qaseem Iqbal

RAG Spider - Web to Markdown Crawler for AI Training Data

lenient_grove/RAG-Spider

Enterprise-grade web crawler that converts messy websites into clean, chunked Markdown for AI systems. Uses Mozilla Readability for 95% cleaner extraction than competitors. Outputs RAG-ready data with metadata and token estimates. Perfect for building knowledge bases and training AI chatbots.

Tejas Rawool

5.0

(1)

universal-web-to-markdown

hachi-dev/universal-web-to-markdown

High-performance tool for AI & RAG pipelines. Converts web pages to clean Markdown by removing noise and fixing relative URLs. Built with Cheerio for extreme speed and low cost ($0.50/1k pages). Perfect for feeding clean data to LLMs.

JI JUN

Compliance-Grade Web Intelligence for AI Agents

ai_solutionist/compliance-web-intel

The scraper AI agents trust. Extract grounded facts with citations, entities, claims & RAG chunks. Built for LangChain, LlamaIndex, AutoGPT. Quality scoring, auto-citations, 6 task modes.