No credit card required
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.
Do you want to learn more about this Actor?
Get a demoThe Actor stopped working from us since it upgraded to 0.3.36
Closed
We are running this actor for production workloads. Please go back to 0.3.35 until you fix the issue
Hello @jorge_rauda,
we're sorry to hear that the latest Actor version stopped working for you.
You can keep using 0.3.35 if you switch to an "untagged" version - simply type in 0.3.35 to the Run Options > Build input option (or pass the Actor version via API / Apify client library).
We'll investigate your issue further in the following days. Thanks for letting us know!
jorge_rauda-owner
We have increased the memory from 1 to 2 GB and it worked again using
I see, our browser-based crawlers have a minimum requirement of 4 GB for a stable and smooth experience. While you'll probably be fine with 2 GB for short runs and simple webpages, we cannot really guarantee a problem-free run with this configuration.
If you're scraping non-JS-based websites, you can always use the Cheerio crawler type, which requires much less memory (works fine with as little as 512 MB) and runs much faster. The obvious downside is that is cannot execute client-side JS and can only be used to scrape static pages.
I'll close this issue now, but feel free to ping up additional questions regarding the performance of this Actor - or anything else.
Cheers!
Apify- 2.8k monthly users
- 434 stars
- 99.9% runs succeeded
- 2.9 days response time
- Created in Mar 2023
- Modified 3 days ago
