Website Content Crawler avatar
Website Content Crawler

Pricing

Pay per usage

Go to Store
Website Content Crawler

Website Content Crawler

Developed by

Apify

Apify

Maintained by Apify

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

4.6 (38)

Pricing

Pay per usage

1310

Total users

49.4k

Monthly users

6.9k

Runs succeeded

>99%

Issue response

3.8 days

Last modified

7 days ago

JA

Crawling is stuck (10h+)

Open

jauns-ai opened this issue
14 days ago

I have set

maxCrawlPages: 500, maxRequestRetries: 3, requestTimeoutSecs: 15,

The max time should be then around 500*3*15 = 22500s (6h)

I have seen this happen in other actor runs, how can I avoid this issue?

jakub.kopecky avatar

Hi, thank you for using Website Content Crawler!

This setting relates to scraping and does not limit the total Actor runtime. To set the maximum Actor runtime, please configure the timeout in the Run options section of the Actor input. When the Actor exceeds this value, it is automatically stopped (even if it hasn't crawled all pages).

Jakub