Website Content Crawler
Pricing
Pay per usage
Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
4.6 (38)
Pricing
Pay per usage
1.1k
Monthly users
6k
Runs succeeded
>99%
Response time
2.3 days
Last modified
7 days ago
Crawler doesn't follow any links in Start URL
I am trying to get Website Content Crawler running with "https://portfoliocharts.com/commentary-all/" as the Start URL - all other settings are default. The crawler simply visits that page and doesn't follow any of the links on it even though it's a very straightforward page with a clear
nhanna
This is the task: https://console.apify.com/view/runs/mcmoAPYoVLxnNIVYs
Hi, thanks for using the Website Content Crawler.
The issue here is that the URLs of the blog posts are not in the same path as your Start URL, which means they weren't matching and thus weren't queued for crawling. You can resolve this by adding a glob pattern in includeUrlGlobs under "Crawler settings" -> "Include URLs (globs)".
For example, in this run, I added the glob https://portfoliocharts.com/[0-9][0-9][0-9][0-9]/[0-9][0-9]/[0-9][0-9]/* to match the blog post URLs, and now the pages are being crawled:
https://console.apify.com/view/runs/sGoTobrZYg08vFYxc
Please try to run the Actor again and let me know if you encounter any issues.
Jakub Kopecky
nhanna
Thank you, rookie mistake!
Pricing
Pricing model
Pay per usageThis Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.
