Pricing

$2.00 / 1,000 delivered results

Try for free

Go to Apify Store

Website Content Crawler — Clean Markdown for AI

Try for free

Crawl public HTML websites into clean Markdown and text for AI, RAG and content research. Export titles, metadata, canonical URLs and links with depth and page limits.

Pricing

$2.00 / 1,000 delivered results

Rating

0.0

(0)

Developer

GUIR Dallel

Actor stats

Bookmarked

Total users

Monthly active users

20 hours ago

Last modified

Categories

Developer tools

You can access the Website Content Crawler — Clean Markdown for AI programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1import { ApifyClient } from 'apify-client';
2
3// Initialize the ApifyClient with your Apify API token
4// Replace the '<YOUR_API_TOKEN>' with your token
5const client = new ApifyClient({
6    token: '<YOUR_API_TOKEN>',
7});
8
9// Prepare Actor input
10const input = {
11    "targets": [
12        "https://books.toscrape.com"
13    ]
14};
15
16// Run the Actor and wait for it to finish
17const run = await client.actor("dallel-data/website-content-to-markdown").call(input);
18
19// Fetch and print Actor results from the run's dataset (if any)
20console.log('Results from dataset');
21console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
22const { items } = await client.dataset(run.defaultDatasetId).listItems();
23items.forEach((item) => {
24    console.dir(item);
25});
26
27// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

Website Content Crawler — Clean Markdown for AI API in JavaScript

The Apify API client for JavaScript is the official library that allows you to use Website Content Crawler — Clean Markdown for AI API in JavaScript or TypeScript, providing convenience functions and automatic retries on errors.

Install the apify-client

$npm install apify-client

Other API clients include:

Website Content Crawler — Clean Markdown for AI API in Python

Website Content Crawler — Clean Markdown for AI API through CLI

Website Content Crawler — Clean Markdown for AI OpenAPI definition

Website Content Crawler — Clean Markdown for AI API

Website Content Scraper: Clean Markdown for AI and RAG

scrapemint/website-content-scraper

Crawl any website and get clean markdown, text, or HTML per page, ready for RAG pipelines, chatbots, and LLM fine tuning. Plain HTTP, no browser, no API key. Pay per page.

Ken M

Website Content Crawler Pro — Markdown for AI & RAG

hipersoft/web-content-crawler-pro

Crawl any website into clean Markdown or text for LLMs, RAG and AI. HTTP + JS-browser modes, main-content (readability) extraction, sitemaps, URL glob filters, cookie-banner removal and CSS selectors. No setup.

hiper soft

Full Website Downloader

schnellscrapers/full-website-downloader

Download a public, server-rendered website as a ZIP with HTML, CSS, JavaScript, images, fonts, and other static assets. Crawl same-host links with configurable depth and limits. Best for landing pages, portfolios, blogs, and documentation; JavaScript-only apps are not rendered.

Nate Schnell

Website Content Crawler - Markdown, 45 Fields

parseforge/website-content-crawler

Crawls websites from start URLs with depth, domain and URL-pattern limits. 45 fields per page: Markdown and plain text, word count, title and meta tags, headings, Open Graph and JSON-LD, author and dates, emails, phones and social links, detected technologies, links and HTTP status. Ready for LLMs.

ParseForge

141

Website Content Crawler for AI — Clean Markdown, 4x Cheaper

joyouscam35875/website-content-crawler

Crawl any website and extract clean text/markdown for LLMs, RAG pipelines, vector databases. BFS crawl with depth control, robots.txt support, boilerplate removal. Perfect for feeding AI models. $0.001/page — 4x cheaper than the official Apify crawler.

Ken Digital

Cloudflare & Anti-Bot Fetcher – Gap-Filler for Crawlers

vietaro/cloudflare-bypass-scraper

Fast HTTP by default, automatic Cloudflare/anti-bot bypass when blocked. The gap-filler for Website Content Crawler, RAG Web Browser, or any crawler that got blocked — retries just their failed pages and returns clean, LLM-ready Markdown.

Vietaro Tools

Website Content Crawler Pro

datascoutapi/website-content-crawler-pro

Crawl websites and extract clean, structured content in Markdown, JSON, or plain text for AI models, LLMs, vector DBs, or RAG pipelines. Fast, reliable, and stealthy, with bulk processing, advanced metadata extraction, and seamless integration with LangChain, LlamaIndex, and AI workflows.

halam

568

3.7

(3)

Website Content Crawler for AI & LLM Data

your_scraper_guy/website-content-crawler-lite

Crawl any website from a seed URL and extract clean Markdown content, ready for LLM training data, RAG pipelines, and vector databases. Set crawl depth, page limits, and domain scope.

Code With Aqib

Web Content Extractor - Clean Markdown for AI

geekguymj/web-content-extractor

Extract clean, readable markdown content from any web page. Removes navigation, ads, footers, and boilerplate — outputs structured markdown optimized for LLM training, RAG pipelines, and AI agents. Pay-per-event pricing. $0.002/page.

Matthew Jenkins

Website Content Crawler: Markdown for AI and RAG

automa-flow/website-content-crawler

Crawl sites you own or are authorized to process into clean Markdown for AI and RAG. Keep headings, tables and code, emit source-linked chunks, and cap pages with HTTP-first limits.

Vadim Bezrukov

Website Contact & Email Extractor

code-node-tools/website-contact-extractor

Crawl a domain or list of URLs and extract emails, phone numbers, and social media handles. Cheerio-based crawling with configurable depth, proxies, and selectable extraction targets.