Website Content Crawler
No credit card required
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
Do you want to learn more about this Actor?
Get a demoAdaptive crawler is failing to crawl the start URL
Open
We are seeing the following error logs in some of our Adaptive Crawler runs:
12024-05-28T18:08:45.004Z ERROR AdaptiveCrawler: Request failed and reached maximum retries. mouse.wheel: Execution context was destroyed, most likely because of a navigation. 22024-05-28T18:08:45.007Z at doScroll (/home/myuser/node_modules/@crawlee/playwright/internals/utils/playwright-utils.js:326:26) 32024-05-28T18:08:45.009Z at async infiniteScroll (/home/myuser/node_modules/@crawlee/playwright/internals/utils/playwright-utils.js:337:9) 42024-05-28T18:08:45.012Z at async genericHandler (/home/myuser/dist/request-handler.js:59:9) 52024-05-28T18:08:45.015Z at async wrap (/home/myuser/node_modules/@apify/timeout/cjs/index.cjs:54:21)
We recently switched to the Adaptive crawler to avoid having to implement our own "fallback" logic of retrying crawls that failed with the Cheerio HttpCrawler. Oddly, the website from this particular crawl usually succeeds with Cheerio so we're not sure why it's using the AdaptiveCrawler.
Please let us know if there's something on our end we can do to fix this as we'd love to reduce the amount of custom logic and instead rely on the adaptive crawler
ApifyActor Metrics
3.9k monthly users
-
718 stars
>99% runs succeeded
2.2 days response time
Created in Mar 2023
Modified 12 hours ago
