Website Content Crawler avatar
Website Content Crawler

Pricing

Pay per usage

Go to Store
Website Content Crawler

Website Content Crawler

Developed by

Apify

Apify

Maintained by Apify

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

4.0 (40)

Pricing

Pay per usage

1392

Total users

53K

Monthly users

7.9K

Runs succeeded

>99%

Issues response

6.8 days

Last modified

4 days ago

UT

Timeout and no data

Open

Autocom opened this issue
16 days ago

Constant timeout or no download of data with a success

VI

vitordcunha

13 days ago

Same here

jiri.spilka avatar

Hi, thank you for using the Website Content Crawler.

It looks like this may have been an intermittent issue, either on the target website’s end or within the Website Content Crawler itself.

I attempted to reproduce the problem, but everything worked. You can see the successful runs here and here.

If you encounter the issue again, please let me know. In the meantime, you might consider increasing the requestTimeoutSecs (e.g., to 60 seconds) and setting maxRequestRetries to a higher value (e.g., 5) to improve reliability.

Best regards, Jiri