Website Content Crawler avatar
Website Content Crawler

Pricing

Pay per usage

Go to Store
Website Content Crawler

Website Content Crawler

Developed by

Apify

Apify

Maintained by Apify

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

4.0 (40)

Pricing

Pay per usage

1392

Total users

53K

Monthly users

7.9K

Runs succeeded

>99%

Issues response

6.8 days

Last modified

4 days ago

DM

Website Content Crawler stuck - cost keeps increasing

Open

digtital_moose opened this issue
3 days ago

Hi,

I aborted the crawler because it got stuck at 527 out of 4.3k links. But the usage costs kept increasing!? So what if i dont keep an eye on it? It just keeps trying indefinitely and i end up with a huge bill without it actually finishing? That would be crazy..

janbuchar avatar

Hello, and thank you for your interest in the Website Content Crawler! It seems like the website that you're scraping actually stopped responding to requests after a period of time (~10 minutes). Judging by the logs, this might actually be caused by a server error on their side. The crawler itself, however, has no way of knowing that, and so it kept retrying the failed requests. In cases like this, it might be warranted to decrease the maxRequestRetries parameter. Also, limiting the maxCrawlPages or setting a timeout on the whole run are good ways to keep the costs in check.