No credit card required
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.
Can we limit the number of pages inside a child?
Open
Often, The websites that are getting crawled are either too big or too small with the depth configuration.
To fix this, We can limit the number of pages but some pages are not getting scrapped due to this.
To handle this, Can we have a new param, say maxPagesPerChild which could allow us to limit of pages a child can add.
Can you please look into this use case? Thank you.
Hello and thank you for your interest in this Actor!
There were some similar feature requests earlier, if I remember correctly. Back then, we backlogged those, because it was just one user's idea - but now that we see that multiple users could benefit from such a feature, we'll increase the priority on this.
However, implementing this requires untrivial changes to the Actor and the way it's tracking the visited pages. We'll look into this and keep you posted here.
Cheers!
Apify- 2.8k monthly users
- 317 stars
- 100.0% runs succeeded
- 4 days response time
- Created in Mar 2023
- Modified 1 day ago
