Website Content Crawler avatar
Website Content Crawler

Pricing

Pay per usage

Go to Store
Website Content Crawler

Website Content Crawler

Developed by

Apify

Apify

Maintained by Apify

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

4.6 (38)

Pricing

Pay per usage

1310

Total users

49.4k

Monthly users

6.9k

Runs succeeded

>99%

Issue response

3.8 days

Last modified

7 days ago

CD

Large number of requests fail

Open

cirez_d opened this issue
a month ago

Hello, lately many requests from runs with the 'cheerio' crawler time out. I am not sure how that is supposed to happen, as the crawler should dynamically update the number of requests and the website does not block the requests (to my understanding). Is there some misconfiguration on my side? I am ok with the run taking some time, it is more important that the requests don't time out, without setting some especially high value. In a test run, I increased the memory and time out duration and it crawled more pages, but still far from all, which surprises me. Any guidance would be very helpful. Thank you.