
Website Content Crawler
Pricing
Pay per usage

Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
4.6 (38)
Pricing
Pay per usage
1310
Total users
49.4k
Monthly users
6.9k
Runs succeeded
>99%
Issue response
3.8 days
Last modified
7 days ago
Is it possible to speed up the processing time?
Open
My average run time for each crawl is about 1 minute. Sometimes it takes less than 30 seconds, or take more than 3 minutes.
Is is possible to make this faster? Ideally around 40 seconds to 1-minute mark

Hey, thanks for using Website Content Crawler!
To speed up crawling, try tweaking the initialConcurrency
Actor input. This should make crawling faster from the start, but overall runtime depends on multiple external factors.
Jakub