Website Content Crawler
No credit card required
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
Do you want to learn more about this Actor?
Get a demoCrawler does not extract content, with no useful logs to debug
Closed
We are attempting to scan our client's website. They have whitelisted US proxies. Unfortunately, most of the time, the results are not extracted. With the same settings, it worked once, but now it completely doesn't. The logs are not insightful enough to resolve or understand the issue the crawler has encountered.
We need to provide an answer to our client on whether the problem can be solved by us or if it's something on their end (e.g., CDN blocking).
Using: CrawlerType - playwright:firefox Proxy - US proxy (3 IP address that our clients have whitelisted)
Hi, thank you for using Website Content Crawler.
I checked the logs of your run and adjusted a couple of settings, and it’s now working. Here are the key changes:
In the logs, I noticed Navigation timed out after 20 seconds. This was due to the Request timeout setting.
Increasing this value to 60 seconds (the default) resolves the issue.
Other settings to change:
1"maxCrawlDepth": 0 -> 20, 2"maxCrawlPages": 0 -> 9999999
- Max crawl depth: Setting this to 0 means the Actor will only scrape the
startUrlswithout further crawling. You’ll need to set it to a higher value; the default is 20. - Max crawl pages: This controls the total number of pages to scrape. Remove this setting to use the default value.
With these updated settings:
1"requestTimeoutSecs": 60, 2"maxCrawlDepth": 20, 3"maxCrawlPages": 9999999
I was able to successfully crawl the website. Here’s a log snippet:
12024-11-12T12:02:44.268Z INFO Crawling will be started using 1 start URLs and 0 sitemap URLs 22024-11-12T12:02:44.907Z INFO PlaywrightCrawler: Starting the crawler. 32024-11-12T12:03:22.368Z INFO Enqueued 66 new links on https://****il/.
Please let me know if it works for you too.
nimble_caretaker
Works well, Thank you for your time.
Apify- 3.8k monthly users
- 636 stars
- 100.0% runs succeeded
- 2.7 days response time
- Created in Mar 2023
- Modified 7 days ago
