Pricing

from $2.00 / 1,000 page crawleds

Website Content Crawler

Crawl websites and extract clean Markdown/text content for RAG pipelines and LLMs. HTTP-first, 10x faster than browser-based crawlers. Guide: https://konabayev.com/tools/website-content-crawler/?utm_source=apify_info&utm_medium=referral&utm_campaign=website-content-crawler

Pricing

from $2.00 / 1,000 page crawleds

Rating

0.0

(0)

Developer

Tugelbay Konabayev

Actor stats

Bookmarked

Total users

Monthly active users

7 hours ago

Last modified

Categories

Developer tools

Start URLs

startUrls

Required

URLs to start crawling from. The crawler follows links on the same domain.

Type:array

Max pages to crawl

maxCrawlPages

Optional

Maximum number of pages to crawl and extract. Increase after trying the free sample.

Type:integer

Minimum:1

Maximum:10000

Default:15

Max crawl depth

maxCrawlDepth

Optional

Maximum link depth from start URL. 0 = only start URLs, 1 = start URLs + their links, etc.

Type:integer

Minimum:0

Maximum:100

Default:2

Output format

outputFormat

Optional

Content output format

Type:string

Default:markdown

Options:

markdowntexthtml

Include URL patterns

includeUrlGlobs

Optional

Only crawl URLs matching these glob patterns (e.g., docs, blog). Empty = crawl all same-domain URLs.

Type:array

Default:

[]

Exclude URL patterns

excludeUrlGlobs

Optional

Skip URLs matching these patterns (e.g., login, admin, *.pdf).

Type:array

Default:

[]

Use sitemap

useSitemap

Optional

Fetch /sitemap.xml and add discovered URLs to the crawl queue.

Type:boolean

Default:false

Max concurrency

maxConcurrency

Optional

Maximum parallel page requests. Higher = faster but more aggressive.

Type:integer

Minimum:1

Maximum:50

Default:20

Page timeout (seconds)

pageTimeout

Optional

Timeout for each page request in seconds.

Type:integer

Minimum:5

Maximum:120

Default:30

Proxy configuration

proxyConfiguration

Optional

Optional proxy settings. Not required for most sites.

Type:object

Website Content Crawler

crawlerbros/website-content-crawler

Crawls websites and extracts clean text, markdown, or HTML content. Ideal for LLM training data, RAG pipelines, and knowledge base building.

Crawler Bros

5.0

Website Content Crawler for AI — Clean Markdown, 4x Cheaper

joyouscam35875/website-content-crawler

Crawl any website and extract clean text/markdown for LLMs, RAG pipelines, vector databases. BFS crawl with depth control, robots.txt support, boilerplate removal. Perfect for feeding AI models. $0.001/page — 4x cheaper than the official Apify crawler.

Ken Digital

AI Content Crawler

kai-agent/ai-content-crawler

Crawl any website and get clean, AI-ready content in markdown format. Perfect for RAG pipelines, LLM training data, and vector database ingestion. Features smart chunking, metadata extraction, and multiple output formats.

Kai Agent

Website to Markdown Crawler â€” AI/RAG Data Pipeline

sovereigntaylor/website-to-markdown

Crawl any website and convert every page to clean, structured Markdown. Perfect for RAG pipelines, LLM training data, vector database ingestion, knowledge base building, and AI-powered search. Extracts main content, strips boilerplate, handles metadata, and chunks output for embeddings. Works with L

Ricardo Akiyoshi

Website Content Crawler

parseforge/website-content-crawler

Crawl any website and pull clean Markdown content ready for AI! Follow links across a whole domain and extract page text, titles, headings, images, and metadata. Perfect for building RAG pipelines, training datasets, knowledge bases, and vector databases. Start crawling content in minutes!

ParseForge

Website to Markdown and RAG Dataset Crawler

orbiscribe/website-rag-dataset-builder

Crawl public websites into clean Markdown, text, metadata, links, JSON-LD, and chunks for RAG and knowledge bases.

Orbiscribe Labs

AI-Ready Website Crawler

optimus-fulcria/ai-ready-website-crawler

Crawl websites and convert to clean markdown for AI/RAG, LLM fine-tuning, and document pipelines.

Fulcria Labs

Stealth Website Crawler

nocturne/stealth-website-crawler

Crawl websites protected by Cloudflare, DataDome, and other anti-bot systems. Extract clean text or markdown for AI/LLM pipelines. Like Website Content Crawler, but for sites that block you.

Nocturne

RAG Web Extractor — Chunked Content for AI Pipelines

junipr/rag-web-extractor

Extract clean markdown from websites for RAG pipelines. Strip nav, ads, boilerplate. Preserve headings, links, images. Recursive crawling with depth control. Chunked output for embedding pipelines. Build AI knowledge bases.

junipr

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.