Pricing

from $0.40 / 1,000 page converted to markdowns

Try for free

Go to Apify Store

Fast Website Content Crawler – Markdown for LLM & RAG

Try for free

Convert any URL, sitemap or whole website to clean Markdown for LLMs, RAG and AI agents. Removes nav, footers and cookie banners, keeps headings, tables and code, adds RAG-ready chunks with token counts, llms.txt generation and an only-changed-pages mode. No browser: fast and cheap.

Pricing

from $0.40 / 1,000 page converted to markdowns

Rating

0.0

(0)

Developer

Cemal Atakli

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

Categories

Developer tools

Agents

You can access the Fast Website Content Crawler – Markdown for LLM & RAG programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1import { ApifyClient } from 'apify-client';
2
3// Initialize the ApifyClient with your Apify API token
4// Replace the '<YOUR_API_TOKEN>' with your token
5const client = new ApifyClient({
6    token: '<YOUR_API_TOKEN>',
7});
8
9// Prepare Actor input
10const input = {
11    "startUrls": [
12        {
13            "url": "https://docs.apify.com/academy/scraping-basics-javascript"
14        },
15        {
16            "url": "https://www.paulgraham.com/greatwork.html"
17        }
18    ],
19    "maxPages": 3
20};
21
22// Run the Actor and wait for it to finish
23const run = await client.actor("gazidev/website-to-markdown").call(input);
24
25// Fetch and print Actor results from the run's dataset (if any)
26console.log('Results from dataset');
27console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
28const { items } = await client.dataset(run.defaultDatasetId).listItems();
29items.forEach((item) => {
30    console.dir(item);
31});
32
33// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

Website to Markdown for LLM & RAG – URL & Sitemap Crawler API in JavaScript

The Apify API client for JavaScript is the official library that allows you to use Fast Website Content Crawler – Markdown for LLM & RAG API in JavaScript or TypeScript, providing convenience functions and automatic retries on errors.

Install the apify-client

$npm install apify-client

Other API clients include:

Fast Website Content Crawler – Markdown for LLM & RAG API in Python

Fast Website Content Crawler – Markdown for LLM & RAG API through CLI

Fast Website Content Crawler – Markdown for LLM & RAG OpenAPI definition

Fast Website Content Crawler – Markdown for LLM & RAG API

Website Content Crawler - Clean Text, Markdown & RAG Chunks

topapi/website-content-crawler

Crawl any website or URL list for $1.20 per 1,000 pages: one row per page with clean text, Markdown, HTML, metadata, links, headings, tables and RAG chunks. Plain HTTP first, headless Chrome only for JavaScript pages. Sitemaps, llms.txt, PDF/DOCX text, Bing/DuckDuckGo search, change monitor.

Top Api

Website to Markdown Scraper for AI & LLMs

quaking_pail/ai-website-content-markdown-scraper

Crawl a website and get clean Markdown of each page's main content, with URL and title. Ready for LLMs, RAG, vector databases and AI agents. Headless Chrome, same-domain crawl, API access.

AI_Builder

946

3.5

(2)

Knowledge Intelligence Engine — Website to Markdown for RAG

ryanclinton/website-content-to-markdown

Turn any website, documentation site or help centre into a retrieval-ready knowledge corpus for RAG and AI search. Clean Markdown plus chunks, change detection, deduplication, retrieval scoring, version awareness and a full corpus audit, in one run.

Ryan Clinton

Web-to-Markdown Generator for AI & RAG Pipelines

profitstack/web-to-markdown-generator-for-ai-rag-pipelines

Convert any website into clean, heading-based chunking, LLM-ready Markdown for RAG and AI agents.

Manas Mantri

Website Content Extractor for RAG: Markdown, HTML, Text

nezha/website-content-crawler

Turn docs sites, help centers, blogs, and websites into clean markdown, text, or HTML for RAG, AI knowledge bases, and internal search. Crawl from start URLs or sitemaps and keep the crawl in scope.

nezha

Website to Markdown for LLMs & RAG — Content Extractor

fetchbase/website-to-markdown

Turn any URL or whole site into clean, LLM-ready Markdown, text, or JSON. Strips nav/ads/boilerplate; keeps headings, links, tables, code. Sitemap-aware concurrent crawl, URL filters, robots.txt respected, rich metadata. Charged only per page extracted — no startup fee.

Fetchbase

Website Content Crawler to Markdown for RAG & LLMs

rod_analytics/website-to-markdown

Low-cost website content crawler: turn any site into clean Markdown for RAG, LLMs and AI agents at a flat $0.30 per 1,000 pages. Main-content extraction, GFM tables, RAG chunks, llms.txt per domain, robots.txt respected.

Rod Services

Website Content Crawler Pro — Markdown for AI & RAG

hipersoft/web-content-crawler-pro

Crawl any website into clean Markdown or text for LLMs, RAG and AI. HTTP + JS-browser modes, main-content (readability) extraction, sitemaps, URL glob filters, cookie-banner removal and CSS selectors. No setup.

Hipersoft

Website Content Crawler

worshipful_knife/website-content-crawler

Deep crawl websites and extract clean text, Markdown, or HTML for LLMs, RAG, and AI apps. Removes navigation, ads, cookie banners. Supports headless browser & HTTP. Sitemap discovery, URL scoping, file downloads. Feed ChatGPT, LangChain, LlamaIndex, Pinecone. The cheapest content crawler on Apify.