Website Content Crawler
No credit card required
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
Do you want to learn more about this Actor?
Get a demoActor timeout
Closed
Getting error that the actor timed out.. Can you help
Hi, thank you for using the Website Content Crawler.
You need to increase the timeout in the Website Content Crawler settings. You can resurrect your run in the console (go to your run, at the bottom there is an option to resurrect) and increase the timeout value to allow the crawler to finish (please see attached screenshot). Currently, your timeout is set to 86,400 seconds. Please try increasing it to, say, 360,000 seconds (the default value).
If you are using the API, you can refer to the documentation here: API Documentation - Resurrect Run.
For future runs, you can update the timeout value by navigating to:
Actors -> Website Content Crawler -> Input -> Run Options -> Timeout (refer to the attached screenshot).
Please let me know if this resolves the issue. I’m happy to provide more details if needed!
shalabi
I have to start over again? It is an expensive crawl
No, no. I apologize, I should have said that with the "resurrect" the run will continue where it left off. It won't start over again.
Regarding the price, I've checked the logs and it seems that the target website is quite slow. I can see a lot of timeouts.
Reclaiming failed requests back to the list or queue. Navigation timed out after 60 seconds.
And the website is quite large.
I’ll go ahead and close this issue now, but please feel free to ask additional questions or raise a new issue.
ApifyActor Metrics
3.9k monthly users
-
718 stars
>99% runs succeeded
2.2 days response time
Created in Mar 2023
Modified 15 hours ago
