Pricing

from $1.20 / 1,000 page crawleds

Try for free

Go to Apify Store

Website Content Crawler - Clean Text, Markdown & RAG Chunks

Try for free

Crawl any website or URL list for $1.20 per 1,000 pages: one row per page with clean text, Markdown, HTML, metadata, links, headings, tables and RAG chunks. Plain HTTP first, headless Chrome only for JavaScript pages. Sitemaps, llms.txt, PDF/DOCX text, Bing/DuckDuckGo search, change monitor.

Pricing

from $1.20 / 1,000 page crawleds

Rating

0.0

(0)

Developer

Top Api

Actor stats

Bookmarked

Total users

Monthly active users

4 days ago

Last modified

Categories

Developer tools

Automation

You can access the Website Content Crawler - Clean Text, Markdown & RAG Chunks programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1import { ApifyClient } from 'apify-client';
2
3// Initialize the ApifyClient with your Apify API token
4// Replace the '<YOUR_API_TOKEN>' with your token
5const client = new ApifyClient({
6    token: '<YOUR_API_TOKEN>',
7});
8
9// Prepare Actor input
10const input = {
11    "startUrls": [
12        {
13            "url": "https://docs.apify.com/academy/scraping-basics-javascript"
14        }
15    ],
16    "maxCrawlPages": 10,
17    "respectRobotsTxtFile": true,
18    "proxyConfiguration": {
19        "useApifyProxy": true
20    }
21};
22
23// Run the Actor and wait for it to finish
24const run = await client.actor("topapi/website-content-crawler").call(input);
25
26// Fetch and print Actor results from the run's dataset (if any)
27console.log('Results from dataset');
28console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
29const { items } = await client.dataset(run.defaultDatasetId).listItems();
30items.forEach((item) => {
31    console.dir(item);
32});
33
34// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

Website to Markdown Crawler API - LLM Text & RAG Chunks API in JavaScript

The Apify API client for JavaScript is the official library that allows you to use Website Content Crawler - Clean Text, Markdown & RAG Chunks API in JavaScript or TypeScript, providing convenience functions and automatic retries on errors.

Install the apify-client

$npm install apify-client

Other API clients include:

Website Content Crawler - Clean Text, Markdown & RAG Chunks API in Python

Website Content Crawler - Clean Text, Markdown & RAG Chunks API through CLI

Website Content Crawler - Clean Text, Markdown & RAG Chunks OpenAPI definition

Website Content Crawler - Clean Text, Markdown & RAG Chunks API

Website Content Crawler — Text, Markdown & HTML for AI/LLM

hichemdev/website-content-crawler

Crawl any website and extract clean text, Markdown, and HTML from every page — ready for LLM, RAG, and AI ingestion.

Hichem Ben Moussa

Website Content Crawler — Text, Titles & Metadata

hipersoft/website-content-crawler

Extract clean, readable content from any list of websites: page title, meta description, headings, main body text, word count and link/image counts. Optional same-domain crawl. Bulk-ready. Great for LLM/RAG ingestion, content audits and research.

hiper soft

Website Content Crawler

jasondev/website-content-crawler

A powerful web crawler that extracts text content from websites, optimized for AI models, Large Language Models (LLMs), vector databases, and Retrieval-Augmented Generation (RAG) pipelines.

Jason Giang

Website Content Extractor for RAG: Markdown, HTML, Text

nezha/website-content-crawler

Turn docs sites, help centers, blogs, and websites into clean markdown, text, or HTML for RAG, AI knowledge bases, and internal search. Crawl from start URLs or sitemaps and keep the crawl in scope.

nezha

Website Content Crawler Pro

datascoutapi/website-content-crawler-pro

Crawl websites and extract clean, structured content in Markdown, JSON, or plain text for AI models, LLMs, vector DBs, or RAG pipelines. Fast, reliable, and stealthy, with bulk processing, advanced metadata extraction, and seamless integration with LangChain, LlamaIndex, and AI workflows.

halam

575

3.7

(3)

Website Content Crawler — Extract Full Site Content

oneary/website-content-crawler

🌐 Full website crawler that extracts structured content (text, headings, metadata, links, images) from any domain. Free platform compute pricing.

Luan M.

Website Content Crawler for AI — Clean Markdown, 4x Cheaper

joyouscam35875/website-content-crawler

Crawl any website and extract clean text/markdown for LLMs, RAG pipelines, vector databases. BFS crawl with depth control, robots.txt support, boilerplate removal. Perfect for feeding AI models. $0.001/page — 4x cheaper than the official Apify crawler.

Ken Digital

Website Content to Markdown for LLM Training

easyapi/website-content-to-markdown-for-llm-training

🚀 Transform web content into clean, LLM-ready Markdown! 📘 Scrape multiple pages, extract main content, and convert to Markdown format. Perfect for AI researchers, data scientists, and LLM developers. Fast, efficient, and customizable. Supercharge your AI training data today! 🌐📝🧠

EasyApi

338

5.0

(2)

Website Content Crawler - Markdown, 45 Fields

parseforge/website-content-crawler

Crawls websites from start URLs with depth, domain and URL-pattern limits. 45 fields per page: Markdown and plain text, word count, title and meta tags, headings, Open Graph and JSON-LD, author and dates, emails, phones and social links, detected technologies, links and HTTP status. Ready for LLMs.

ParseForge

176

Website Content Crawler – Markdown for AI, Emails & Contacts

locomint/website-crawler-content-contacts

Website content crawler that turns a whole site into clean Markdown for LLM and RAG pipelines, with the role emails, social profiles and WhatsApp numbers found on each page. Seeded from the sitemap, honours robots.txt, plain HTTP with no browser.