Website Content Crawler avatar
Website Content Crawler

Pricing

Pay per usage

Go to Store
Website Content Crawler

Website Content Crawler

Developed by

Apify

Apify

Maintained by Apify

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

4.6 (38)

Pricing

Pay per usage

1310

Total users

49.4k

Monthly users

6.9k

Runs succeeded

>99%

Issue response

3.8 days

Last modified

7 days ago

SS

Is it possible to speed up the processing time?

Open

sheldon-supreme opened this issue
21 days ago

My average run time for each crawl is about 1 minute. Sometimes it takes less than 30 seconds, or take more than 3 minutes.

Is is possible to make this faster? Ideally around 40 seconds to 1-minute mark

jakub.kopecky avatar

Hey, thanks for using Website Content Crawler!

To speed up crawling, try tweaking the initialConcurrency Actor input. This should make crawling faster from the start, but overall runtime depends on multiple external factors.

Jakub