Website Content Crawler
Pricing
Pay per usage
Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
4.6 (38)
Pricing
Pay per usage
1224
Monthly users
6.4k
Runs succeeded
>99%
Response time
4.6 days
Last modified
2 days ago
the crawler stops half way through the crawling process
Closed
When crawling large websites (e.g., 2000+ pages), the crawler sometimes fails on a page with this error:
Request failed and reached maximum retries. Error: Resource ... is not available in the format requested by the Accept header.
This causes the entire crawl to stop because it can’t proceed to the next page. However, if I start the crawl from the failed page, it works and continues normally—until it hits another page with the same issue.
Problem: A single page failure stops the whole crawl. The error is intermittent and not tied to specific URLs. Retrying the whole crawl is inefficient and unreliable making a "complete crawl job" impossible without stiching the jobs manually. WhatI need: • Ability to retry failed requests individually, manually after the crawler stops, possibly with a different proxy or headers. •. Resumable crawling or checkpoint support. Please advise how to fix this or if there’s a workaround we can implement.
Hi,
It looks like there’s an issue with the target website. The problem is that you're crawling paginated pages, so if one of them fails, the subsequent ones aren’t crawled either.
There’s no perfect solution here, but increasing the requestTimeoutSecs and maxRequestRetries values can help.
In my configuration, I used "maxRequestRetries": 20 and "requestTimeoutSecs": 120, and the run completed with 0 failed requests.
I hope this helps! I’ll close the issue for now, but feel free to ask more questions or reopen it if needed. Jiri
Pricing
Pricing model
Pay per usageThis Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.
