
Website Content Crawler
Pricing
Pay per usage

Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
3.7 (41)
Pricing
Pay per usage
1501
Total users
58K
Monthly users
8.1K
Runs succeeded
>99%
Issues response
7.6 days
Last modified
3 hours ago
Didn't crawl the entire page and seemed to do it in no particular orer
Closed
I an trying to crawl https://vcnewsdaily.com/ and pull up all the reports on there, but when I ran it it didn't crawl all of them and seemed to do it out of order. Can you please assist?
Hello, and thank you for your interest in this Actor!
Since the Actor processes the requests in parallel, it doesn't guarantee that the order of the results will be the same as the order of the links on the initial page.
As for the incomplete results, you are setting the maxResults
input option to 10
, so the Actor will only crawl 10 pages before finishing. You can see this in the Actor log too:
2025-05-27T19:07:29.146Z INFO Reached max results (10), ....
I'll close this issue now, but feel free to ask additional questions if you have any.
Cheers!