Website Content Crawler
Pricing
Pay per usage
Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
3.9 (41)
Pricing
Pay per usage
1537
Total users
59K
Monthly users
7.9K
Runs succeeded
>99%
Issues response
7.8 days
Last modified
2 days ago
Actor timeout
Closed
Getting error that the actor timed out.. Can you help
Hi, thank you for using the Website Content Crawler.
You need to increase the timeout in the Website Content Crawler settings. You can resurrect your run in the console (go to your run, at the bottom there is an option to resurrect) and increase the timeout value to allow the crawler to finish (please see attached screenshot). Currently, your timeout is set to 86,400 seconds. Please try increasing it to, say, 360,000 seconds (the default value).
If you are using the API, you can refer to the documentation here: API Documentation - Resurrect Run.
For future runs, you can update the timeout value by navigating to:
Actors -> Website Content Crawler -> Input -> Run Options -> Timeout (refer to the attached screenshot).
Please let me know if this resolves the issue. I’m happy to provide more details if needed!
shalabi
I have to start over again? It is an expensive crawl
No, no. I apologize, I should have said that with the "resurrect" the run will continue where it left off. It won't start over again.
Regarding the price, I've checked the logs and it seems that the target website is quite slow. I can see a lot of timeouts.
And the website is quite large.
I’ll go ahead and close this issue now, but please feel free to ask additional questions or raise a new issue.
