Website Content Crawler avatar
Website Content Crawler

Pricing

Pay per usage

Go to Store
Website Content Crawler

Website Content Crawler

apify/website-content-crawler

Developed by

Apify

Maintained by Apify

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

4.6 (38)

Pricing

Pay per usage

1.1k

Monthly users

5.9k

Runs succeeded

>99%

Response time

2.3 days

Last modified

6 days ago

MC

Crawl hung at finished

Open
mcantrell opened this issue
2 months ago

The crawler appeared to finish but the container did not shut down. I had to abort after 5m of idle time after the 'Finished' message.

jiri.spilka avatar

Hi, thank you for using the Website Content Crawler.

I'm not entirely sure what happened in your case. I tried to reproduce the issue using the same configuration (and also a different one), but I was not successful (meaning that container was stopped).

You can always set up a timeout for a run. It seems you've already adjusted it to 43,200 seconds. However, it can be tricky to configure since it's difficult to predict exactly how long a run will take. I’ll forward this to my colleague for further investigation. I’ll also ask customer support to reimburse you, as this shouldn’t happen.

I’ll leave this issue open for now. Jiri

MS

mykola_scrapes

2 months ago

Hi Jiri,

I'm experiencing the same issue.

https://console.apify.com/view/runs/mN9hlbWLuabOn6pu4

Also I got billed for the full 5 minutes of execution

Expected result: The run is finished after first page is extracted.

Actual result: Run status is in progress for at least few minutes after the page is extracted.

Pricing

Pricing model

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.