Website Content Crawler
No credit card required
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
Do you want to learn more about this Actor?
Get a demoCan I limit the max number of pages to crawl for each start URL?
Closed
After setting 10 home pages as start URLs and the max number of pages to crawl, it seems some domain got more pages than others.
Is there a way to make the crawling more even by like setting the max pages for each domain?
Hi, thank you for trying Website Content Crawler.
Unfortunately, it’s not possible to control the number of crawled pages for each startURL. When crawling starts, the startURLs are always processed first. As new URLs are discovered, they are added to the request queue and processed in order, without a way to limit this per startURL.
However, you can address this by running the Website Content Crawler separately for each domain and setting the maximum number of crawled pages individually.
I’ll go ahead and close this issue now. If you have further questions, feel free to open a new issue or ask again.
ApifyActor Metrics
3.9k monthly users
-
718 stars
>99% runs succeeded
2.2 days response time
Created in Mar 2023
Modified 15 hours ago
