Website Content Crawler
No credit card required
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
Do you want to learn more about this Actor?
Get a demoCrawling does not work for some sitemaps
Open
Hi team,
I've been trying to crawl this sitemap: https://www.ynab.com/sitemap.xml but the actor is failing with no outputs.
I was unable to find the results but this was working previously as I had the data from it.
Can you please look at it?
Hello, Thank you for trying Website Content Crawler. We will investigate the issue with that sitemap further; in the meantime, I found a workaround for your use case. You can upload a text file with the sitemap, and then you can crawl through it. Best regards,
Hello, Just to give a follow-up on this, we found the issue, and we will fix it. You can monitor it here: https://github.com/apify/crawlee/issues/2698 Best regards,
The issue has been fixed and deployed. Let me close this issue now.
sai_sampath
I have retested with both default version and the beta version, It's still not working; Please find the same below.
https://console.apify.com/actors/runs/qgd42UHL462Zs8nwU https://console.apify.com/actors/runs/ru3hLbZXxRvYKFgIm
Yes, you’re correct—the sitemap isn’t loading properly for this particular website. We’ll look into the issue.
In the meantime, you can start crawling the website using https://www.ynab.com as the start URL. With this adjustment, I was able to crawl successfully.
12024-10-22T14:29:25.931Z INFO AdaptiveCrawler: Running browser request handler for https://www.ynab.com/ 22024-10-22T14:29:54.061Z INFO Enqueued 42 new links on https://www.ynab.com/. 32024-10-22T14:29:54.247Z WARN Aborting the sitemap loading after being stuck for 30 seconds. 42024-10-22T14:29:54.591Z INFO AdaptiveCrawler: Running browser request handler for https://www.ynab.com/about-us 52024-10-22T14:29:54.594Z INFO AdaptiveCrawler: Running browser request handler for https://www.ynab.com/careers 62024-10-22T14:29:55.138Z INFO AdaptiveCrawler: Running browser request handler for https://www.ynab.com/press 72024-10-22T14:29:55.360Z INFO AdaptiveCrawler: Running browser request handler for https://www.ynab.com/ynab-the-book 82024-10-22T14:30:25.029Z INFO AdaptiveCrawler:Statistics: AdaptiveCrawler request statistics: {"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":28180,"requestsFinishedPerMinute":1,"requestsFailedPerMinute":0,"requestTotalDurationMillis":28180,"requestsTotal":1,"crawlerRuntimeMillis":61021,"retryHistogram":[1]} 92024-10-22T14:30:25.032Z INFO AdaptiveCrawler:AutoscaledPool: state {"currentConcurrency":4,"desiredConcurrency":3,"systemStatus":{"isSystemId... [trimmed]
Apify- 3.8k monthly users
- 635 stars
- 100.0% runs succeeded
- 2.7 days response time
- Created in Mar 2023
- Modified 7 days ago
