Pricing

from $3.00 / 1,000 delivered pages

Try for free

Go to Apify Store

Website Content Crawler | Bulk Text & Markdown

Try for free

Fetch public website pages in bulk and export clean text, titles, Markdown and source evidence, with bounded concurrency, robots checks and explicit partial results.

Pricing

from $3.00 / 1,000 delivered pages

Rating

0.0

(0)

Developer

tingyou333 zhuang

Actor stats

Bookmarked

Total users

Monthly active users

13 days ago

Last modified

Categories

Developer tools

Start URLs

startUrls

Required

Bare domains, IDNs, or full public HTTP(S) URLs. One supplied page per entry.

Type:string[] | null

Min. items:0

Max. items:100000

Default:

null

maxConcurrency

Optional

Resource safety limit; reached limits are reported explicitly.

Type:integer

Minimum:1

Maximum:64

Default:16

maxRequests

Optional

Resource safety limit; reached limits are reported explicitly.

Type:integer

Minimum:1

Maximum:1000000

Default:30000

maxPages

Optional

Resource safety limit; reached limits are reported explicitly.

Type:integer

Minimum:1

Maximum:100000

Default:10000

maxBodyBytes

Optional

Resource safety limit; reached limits are reported explicitly.

Type:integer

Minimum:1024

Maximum:10000000

Default:2000000

maxTextChars

Optional

Resource safety limit; reached limits are reported explicitly.

Type:integer

Minimum:100

Maximum:2000000

Default:500000

maxRedirects

Optional

Resource safety limit; reached limits are reported explicitly.

Type:integer

Minimum:0

Maximum:10

Default:5

pageTimeoutSecs

Optional

Resource safety limit; reached limits are reported explicitly.

Type:integer

Minimum:1

Maximum:120

Default:40

runTimeoutSecs

Optional

Resource safety limit; reached limits are reported explicitly.

Type:integer

Minimum:1

Maximum:3600

Default:240

Clash fakeIP DoH fallback

fakeIpDohFallback

Optional

Only all-fakeIP OS DNS uses fixed public Cloudflare DoH, validated answers, pinned TCP and hostname TLS. No fallback for other DNS failures.

Type:boolean

Default:true

Fast Website Content Crawler

6sigmag/fast-website-content-crawler

A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

4.7K

4.9

(7)

Deep URL Content Crawler

6sigmag/deep-url-content-crawler

Scrape Failed Killer! A high-performance web scraper that rapidly extracts and analyzes content from multiple URLs simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

115

Website Content Crawler - Clean Text, Markdown & RAG Chunks

topapi/website-content-crawler

Crawl any website or URL list for $1.20 per 1,000 pages: one row per page with clean text, Markdown, HTML, metadata, links, headings, tables and RAG chunks. Plain HTTP first, headless Chrome only for JavaScript pages. Sitemaps, llms.txt, PDF/DOCX text, Bing/DuckDuckGo search, change monitor.

Top Api

Deep Website Content Crawler

6sigmag/deep-website-content-crawler

Scrape Failed Killer! A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

1.1K

3.0

(1)

Website Content Crawler — Text, Markdown & AI Chunks

agency-shift/web-content-crawler

Extract text or clean Markdown from public HTML websites. Create source-linked sections for AI knowledge bases while keeping one row per page. Preserve headings, code and tables; inspect a crawl report for failed URLs and truncated content. No JavaScript rendering.

Mako

Blog / Dated Content Crawler

diarmuidr/blog-content-crawler

Crawl an entire blog / knowledge base or filter to just the new content. Supporting relevant AI queries by filtering pages by date

Diarmuid

221

5.0

(1)

Web Crawler & Semantic Schema-Enhanced Extractor

devil_port369-owner/web-crawler

Depth-controlled web crawler that transforms websites into structured analytics-ready data. Starting from one or more URLs, it crawls internal links up to a configurable depth and outputs detailed JSON records per page

DataFusionX

5.0

(8)

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.

Hounderd

Website Content Crawler Pro

datascoutapi/website-content-crawler-pro

Crawl websites and extract clean, structured content in Markdown, JSON, or plain text for AI models, LLMs, vector DBs, or RAG pipelines. Fast, reliable, and stealthy, with bulk processing, advanced metadata extraction, and seamless integration with LangChain, LlamaIndex, and AI workflows.

halam

575

3.7

(3)

Website Content Crawler - Markdown, 45 Fields

parseforge/website-content-crawler

Crawls websites from start URLs with depth, domain and URL-pattern limits. 45 fields per page: Markdown and plain text, word count, title and meta tags, headings, Open Graph and JSON-LD, author and dates, emails, phones and social links, detected technologies, links and HTTP status. Ready for LLMs.