Under maintenance

Pricing

from $0.70 / 1,000 page scrapeds

Try for free

Go to Apify Store

Website Content Crawler

Under maintenance

Try for free

Deep crawl websites and extract clean text, Markdown, or HTML for LLMs, RAG, and AI apps. Removes navigation, ads, cookie banners. Supports headless browser & HTTP. Sitemap discovery, URL scoping, file downloads. Feed ChatGPT, LangChain, LlamaIndex, Pinecone. The cheapest content crawler on Apify.

Pricing

from $0.70 / 1,000 page scrapeds

Rating

0.0

(0)

Developer

kata Kuri

Actor stats

Bookmarked

Total users

Monthly active users

3 months ago

Last modified

Categories

Developer tools

Automation

Start URLs

startUrls

Required

One or more URLs to start crawling from. The crawler will follow links found on these pages.

Type:array

Crawler type

crawlerType

Optional

Select the crawler engine. Cheerio (HTTP) is ~10x faster — use it for static sites and docs. Playwright (browser) only when JavaScript rendering is required.

Type:string

Default:playwright:adaptive

Options:

playwright:adaptiveplaywright:firefoxcheerio

Max pages

maxCrawlPages

Optional

Maximum number of pages to crawl. NOTE: Synchronous API endpoints have a 300s hard limit. With Playwright, you can crawl ~25 pages in 300s; with Cheerio, ~200+. Increase 'Run options > Timeout' for larger crawls when running asynchronously.

Type:integer

Minimum:1

Maximum:1000000

Default:25

Max crawling depth

maxCrawlDepth

Optional

Maximum link depth from start URLs. 0 = only start URLs.

Type:integer

Minimum:0

Maximum:100

Default:20

Max concurrency

maxConcurrency

Optional

Maximum number of pages crawled in parallel.

Type:integer

Minimum:1

Maximum:200

Default:10

Exclude URLs (globs)

excludeUrlGlobs

Optional

Glob patterns for URLs to skip. E.g. https://example.com/archive/**

Type:array

Default:

[]

Include URLs (globs)

includeUrlGlobs

Optional

If set, only URLs matching these globs will be crawled.

Type:array

Default:

[]

Output format

outputFormat

Optional

Format of the extracted content.

Type:string

Default:markdown

Options:

textmarkdownhtml

Additional elements to remove (CSS selector)

removeElementsCssSelector

Optional

CSS selector for additional elements to remove. Nav, footer, header, ads, scripts, and iframes are always removed by default.

Type:string

Default:

Extract HTML elements (CSS selector)

extractElementsCssSelector

Optional

If set, only keep content from elements matching this selector. E.g. 'article, main, .content'

Type:string

Default:

Remove cookie warnings

removeCookieWarnings

Optional

Attempt to auto-dismiss cookie consent banners.

Type:boolean

Default:true

Wait for dynamic content (ms)

waitForDynamicContent

Optional

Time in ms to wait after page load for dynamic content. Only for browser crawler.

Type:integer

Minimum:0

Maximum:30000

Default:1000

Maximum scroll height (px)

maxScrollHeight

Optional

Scroll the page to this height to trigger lazy loading. 0 = no scroll.

Type:integer

Minimum:0

Maximum:100000

Default:0

Expand clickable elements

expandClickables

Optional

Click elements like accordions and tabs to expand their content.

Type:boolean

Default:false

Save files

saveFiles

Optional

Download linked document files (PDF, DOC, DOCX, XLS, XLSX, CSV).

Type:boolean

Default:false

Use sitemaps

useSitemaps

Optional

Discover additional URLs from sitemaps.

Type:boolean

Default:true

Proxy configuration

proxyConfiguration

Optional

Proxy settings for the crawler.

Type:object

Default:

{
  "useApifyProxy": true
}

Request timeout (seconds)

requestTimeoutSecs

Optional

Maximum time for loading a single page. Lower values help avoid hanging on blocked sites.

Type:integer

Minimum:10

Maximum:300

Default:30

Website Content Crawler

mikolabs/website-content-crawler

Deep-crawl websites to extract clean text, Markdown, or HTML for AI/LLM apps, RAG pipelines, and vector databases. Supports adaptive crawling, HTML cleaning, file downloads, and structured dataset output. Easily integrates with LangChain, LlamaIndex, and other LLM tools.

Mikolabs

5.0

Website Content Crawler - Markdown & Text for LLM / RAG

pear_fight/website-content-crawler-markdown-text-for-llm-rag

Crawl any website and extract clean article text and Markdown, ready to feed into LLMs, ChatGPT, vector databases and RAG pipelines. Removes navigation, ads and boilerplate. Configurable crawl depth and page limits. Export to JSON, CSV, Excel.

Harald

Website Content Crawler

myagizm/website-content-crawler

Crawl any website and extract clean, LLM-ready Markdown for RAG pipelines, vector databases and AI agents. Boilerplate, navigation and cookie banners removed. No browser, no login.

MYM

Website Content Crawler Pro

datascoutapi/website-content-crawler-pro

Crawl websites and extract clean, structured content in Markdown, JSON, or plain text for AI models, LLMs, vector DBs, or RAG pipelines. Fast, reliable, and stealthy, with bulk processing, advanced metadata extraction, and seamless integration with LangChain, LlamaIndex, and AI workflows.

halam

566

3.7

Website Content Crawler

apify/website-content-crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

Apify

151K

4.6

Website Content Crawler

rupom888/website-content-crawler

Syed Rupom

Website Content Crawler Pro — Markdown for AI & RAG

hipersoft/web-content-crawler-pro

Crawl any website into clean Markdown or text for LLMs, RAG and AI. HTTP + JS-browser modes, main-content (readability) extraction, sitemaps, URL glob filters, cookie-banner removal and CSS selectors. No setup.

hiper soft

Website Content Crawler — Text, Markdown & HTML for AI/LLM

hichemdev/website-content-crawler

Crawl any website and extract clean text, Markdown, and HTML from every page — ready for LLM, RAG, and AI ingestion.

Hichem Ben Moussa

AI Web Content Crawler - Markdown for LLMs

intelscrape/ai-web-content-crawler

Crawl any website and extract clean Markdown optimized for LLM training, RAG pipelines, and AI knowledge bases - removes boilerplate and outputs structured JSON with URL, title, markdown, and metadata.

IntelScrape

Website Crawler API — Markdown for RAG

tugelbay/website-content-crawler

Website crawler API for public pages and clean Markdown, text, or HTML output for RAG pipelines, AI agents, documentation indexing, and monitoring. Guide: https://konabayev.com/tools/website-content-crawler/?utm_source=apify_info&utm_medium=referral&utm_campaign=website-content-crawler