Pricing

from $1.00 / 1,000 pages

Go to Apify Store

Website Content Crawler — Text, Markdown & HTML for AI/LLM

Try for free

Crawl any website and extract clean text, Markdown, and HTML from every page — ready for LLM, RAG, and AI ingestion.

Pricing

from $1.00 / 1,000 pages

Rating

0.0

(0)

Developer

Hichem Ben Moussa

Actor stats

Bookmarked

Total users

Monthly active users

11 days ago

Last modified

Website Content Crawler - Markdown for AI, LLM & RAG API through CLI

The Apify CLI is the official tool that allows you to use Website Content Crawler — Text, Markdown & HTML for AI/LLM locally, providing convenience functions and automatic retries on errors.

Install Apify CLI

Using installation script (macOS/Linux):

$curl -fsSL https://apify.com/install-cli.sh | bash

Using installation script (Windows):

$irm https://apify.com/install-cli.ps1 | iex

Using Homebrew:

$brew install apify-cli

Using npm:

$npm install -g apify-cli

Other API clients include:

Website Content Crawler — Text, Markdown & HTML for AI/LLM API in Python

Website Content Crawler — Text, Markdown & HTML for AI/LLM API in JavaScript

Website Content Crawler — Text, Markdown & HTML for AI/LLM OpenAPI definition

Website Content Crawler — Text, Markdown & HTML for AI/LLM API

Website to Markdown Crawler for LLM & RAG

logiover/website-text-markdown-crawler

Crawl any website to clean Markdown and plain text for LLM training and RAG. HTML to Markdown, no API or login. Export website text to CSV or JSON.

Logiover

Website Content Scraper: Clean Markdown for AI and RAG

scrapemint/website-content-scraper

Crawl any website and get clean markdown, text, or HTML per page, ready for RAG pipelines, chatbots, and LLM fine tuning. Plain HTTP, no browser, no API key. Pay per page.

Ken M

Website to Markdown Scraper

receptional_blender/website-to-markdown-scraper

Crawl any website and turn its pages into clean Markdown — plus optional plain text, raw HTML and full-page screenshots. Built for LLM, RAG and AI training datasets.

Assia Fadli

Site Crawler: Website → Markdown Corpus for LLM/RAG

boxbox10/site-crawler

Crawl a whole website or docs site and get one clean, LLM-ready Markdown + JSON record per page (title, headings, content, links, token count). Built for RAG ingestion and AI knowledge bases.

Marvin Eguilos

Website Content Crawler for AI & LLM Data

your_scraper_guy/website-content-crawler-lite

Crawl any website from a seed URL and extract clean Markdown content, ready for LLM training data, RAG pipelines, and vector databases. Set crawl depth, page limits, and domain scope.

Code With Aqib

Website to RAG Markdown Crawler

knotted_tussock/rag-markdown-crawler

Crawl any website or docs site and export clean Markdown plus JSONL-style chunks for RAG, LLM apps, and AI agents.

Ralph T

AI Content Crawler

kai-agent/ai-content-crawler

Crawl any website and get clean, AI-ready content in markdown format. Perfect for RAG pipelines, LLM training data, and vector database ingestion. Features smart chunking, metadata extraction, and multiple output formats.

Kai Agent

Website Content Crawler — AI & RAG Ready

santamaria-automations/website-content-crawler

Crawl any website and extract clean Markdown and plain text optimized for AI ingestion, RAG pipelines, and LLM context. Readability-style main content extraction removes ads, navs, and footers. Configurable depth, concurrency, and page limits. Pay-per-page.

NanoScrape

Website Content Crawler - Markdown & Text for LLM / RAG

pear_fight/website-content-crawler-markdown-text-for-llm-rag

Crawl any website and extract clean article text and Markdown, ready to feed into LLMs, ChatGPT, vector databases and RAG pipelines. Removes navigation, ads and boilerplate. Configurable crawl depth and page limits. Export to JSON, CSV, Excel.