You can access the Website Content Crawler — Text, Markdown & HTML for AI/LLM programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1import { ApifyClient } from 'apify-client';
2
3// Initialize the ApifyClient with your Apify API token
4// Replace the '<YOUR_API_TOKEN>' with your token
5const client = new ApifyClient({
6    token: '<YOUR_API_TOKEN>',
7});
8
9// Prepare Actor input
10const input = {
11    "startUrls": [
12        {
13            "url": "https://apify.com"
14        }
15    ],
16    "crawlerType": "cheerio",
17    "maxCrawlPages": 10,
18    "maxCrawlDepth": 1,
19    "includeGlobs": [],
20    "excludeGlobs": []
21};
22
23// Run the Actor and wait for it to finish
24const run = await client.actor("hichemdev/website-content-crawler").call(input);
25
26// Fetch and print Actor results from the run's dataset (if any)
27console.log('Results from dataset');
28console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
29const { items } = await client.dataset(run.defaultDatasetId).listItems();
30items.forEach((item) => {
31    console.dir(item);
32});
33
34// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

Website Content Crawler - Markdown for AI, LLM & RAG API in JavaScript

The Apify API client for JavaScript is the official library that allows you to use Website Content Crawler — Text, Markdown & HTML for AI/LLM API in JavaScript or TypeScript, providing convenience functions and automatic retries on errors.

Install the apify-client

$npm install apify-client

Other API clients include:

Website Content Crawler — Text, Markdown & HTML for AI/LLM API in Python

Website Content Crawler — Text, Markdown & HTML for AI/LLM API through CLI

Website Content Crawler — Text, Markdown & HTML for AI/LLM OpenAPI definition

Website Content Crawler — Text, Markdown & HTML for AI/LLM API

Website to Markdown Crawler for LLM & RAG

logiover/website-text-markdown-crawler

Crawl any website to clean Markdown and plain text for LLM training and RAG. HTML to Markdown, no API or login. Export website text to CSV or JSON.

Logiover

Website Content Scraper: Clean Markdown for AI and RAG

scrapemint/website-content-scraper

Crawl any website and get clean markdown, text, or HTML per page, ready for RAG pipelines, chatbots, and LLM fine tuning. Plain HTTP, no browser, no API key. Pay per page.

Ken M

Website to Markdown Scraper

receptional_blender/website-to-markdown-scraper

Crawl any website and turn its pages into clean Markdown — plus optional plain text, raw HTML and full-page screenshots. Built for LLM, RAG and AI training datasets.

Assia Fadli

Site Crawler: Website → Markdown Corpus for LLM/RAG

boxbox10/site-crawler

Crawl a whole website or docs site and get one clean, LLM-ready Markdown + JSON record per page (title, headings, content, links, token count). Built for RAG ingestion and AI knowledge bases.

Marvin Eguilos

Website Content Crawler for AI & LLM Data

your_scraper_guy/website-content-crawler-lite

Crawl any website from a seed URL and extract clean Markdown content, ready for LLM training data, RAG pipelines, and vector databases. Set crawl depth, page limits, and domain scope.

Code With Aqib

Website Content Crawler - Markdown, 45 Fields

parseforge/website-content-crawler

Crawls websites from starting URLs and returns each page's full text content as Markdown and cleaned HTML, ready for AI and LLM pipelines.

ParseForge

138

Website Content Crawler - Markdown, 45 Fields

excitable_pickle/web-markdown-rag-crawler

Crawls websites from starting URLs and returns each page's full text content as Markdown and cleaned HTML, ready for AI and LLM pipelines.

Pavan

5.0

Website to Markdown API & Crawler – RAG/LLM

signal_lab/website-to-markdown-crawler

Website-to-Markdown API and crawler for RAG and LLM datasets. Extract clean Markdown, SEO metadata, links, JSON-LD, sitemaps, and robots.txt-compliant pages.

Signal Lab

Website to RAG Markdown Crawler

knotted_tussock/rag-markdown-crawler

Crawl any website or docs site and export clean Markdown plus JSONL-style chunks for RAG, LLM apps, and AI agents.

Ralph T

Website Content Crawler — AI & RAG Ready

santamaria-automations/website-content-crawler

Crawl any website and extract clean Markdown and plain text optimized for AI ingestion, RAG pipelines, and LLM context. Readability-style main content extraction removes ads, navs, and footers. Configurable depth, concurrency, and page limits. Pay-per-page.