Website Content Crawler
Pricing
Pay per usage
Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
4.6 (38)
Pricing
Pay per usage
1.1k
Monthly users
6k
Runs succeeded
>99%
Response time
2.3 days
Last modified
7 days ago
"htmlUrl" points to the same URL for multiple pages
Hello, We have encountered the following issue in the actor run:
- Some pages have identical HTML URLs (htmlUrl), resulting in duplicated content and missing the actual results.
- Upon reviewing the dataset, I noticed that the htmlUrls are recorded as the same, even though each affected and crawled URL contains different content.
- Could it be that the URLs are being truncated at some point during the saving process due to their length, causing the htmlUrl to appear duplicated?
You can Ctrl-F for this htmlUrl (please delete after you viewed this)
The status code is 200, loaded/crawled Url is identical for each page, but the content for htmlUrl is the same across pages.
This is currently a significant issue for us. I would greatly appreciate your prompt support and any suggestions for a temporary workaround.
Thank you!
Hi, thank you for using the Website Content Crawler.
There is an issue with key (URL) truncation when saving into the key-value store that affects URLs longer than 145 characters. This will be fixed, and I will keep you updated.
Jakub
Hi,
The fix was released in the beta build of the Actor.
Please try running it with the beta build and let me know.
Jakub
Pricing
Pricing model
Pay per usageThis Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.
