Website Content Crawler avatar
Website Content Crawler
Try for free

No credit card required

View all Actors
Website Content Crawler

Website Content Crawler

apify/website-content-crawler
Try for free

No credit card required

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.

User avatar

Adaptive crawler is failing to crawl the start URL

Open

embrace_ai opened this issue
a month ago

We are seeing the following error logs in some of our Adaptive Crawler runs:

12024-05-28T18:08:45.004Z ERROR AdaptiveCrawler: Request failed and reached maximum retries. mouse.wheel: Execution context was destroyed, most likely because of a navigation.
22024-05-28T18:08:45.007Z     at doScroll (/home/myuser/node_modules/@crawlee/playwright/internals/utils/playwright-utils.js:326:26)
32024-05-28T18:08:45.009Z     at async infiniteScroll (/home/myuser/node_modules/@crawlee/playwright/internals/utils/playwright-utils.js:337:9)
42024-05-28T18:08:45.012Z     at async genericHandler (/home/myuser/dist/request-handler.js:59:9)
52024-05-28T18:08:45.015Z     at async wrap (/home/myuser/node_modules/@apify/timeout/cjs/index.cjs:54:21)

We recently switched to the Adaptive crawler to avoid having to implement our own "fallback" logic of retrying crawls that failed with the Cheerio HttpCrawler. Oddly, the website from this particular crawl usually succeeds with Cheerio so we're not sure why it's using the AdaptiveCrawler.

Please let us know if there's something on our end we can do to fix this as we'd love to reduce the amount of custom logic and instead rely on the adaptive crawler

Developer
Maintained by Apify
Actor metrics
  • 2.8k monthly users
  • 122 stars
  • 99.9% runs succeeded
  • 3.9 days response time
  • Created in Mar 2023
  • Modified 6 days ago