No credit card required
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.
Error: incorrect header check
Closed
A recent run failed multiple times with this error. Here's a relevant log snippet:
12024-05-13T18:32:30.260Z ACTOR: Pulling Docker image of build Q0D3SU0nCpLVGrtw4 from repository. 22024-05-13T18:32:30.383Z ACTOR: Creating Docker container. 32024-05-13T18:32:30.511Z ACTOR: Starting Docker container. 42024-05-13T18:32:31.818Z Starting X virtual framebuffer using: Xvfb :99 -ac -screen 0 1920x1080x24+32 -nolisten tcp 52024-05-13T18:32:31.826Z Executing main command 62024-05-13T18:32:35.912Z INFO System info {"apifyVersion":"3.1.16","apifyClientVersion":"2.9.0","crawleeVersion":"3.8.1","osType":"Linux","nodeVersion":"v18.19.1"} 72024-05-13T18:32:36.139Z INFO Discovering possible sitemap files from the start URLs... 82024-05-13T18:32:38.358Z node:events:495 92024-05-13T18:32:38.360Z throw er; // Unhandled 'error' event 102024-05-13T18:32:38.362Z ^ 112024-05-13T18:32:38.363Z 122024-05-13T18:32:38.365Z Error: incorrect header check 132024-05-13T18:32:38.366Z at Zlib.zlibOnError [as onerror] (node:zlib:189:17) 142024-05-13T18:32:38.368Z at Zlib.callbackTrampoline (node:internal/async_hooks:128:17) 152024-05-13T18:32:38.370Z Emitted 'error' event on Gunzip instance at: 162024-05-13T18:32:38.371Z at Gunzip.onerror (node:internal/streams/readable:828:14) 172024-05-13T18:32:38.373Z at Gunzip.emit (node:events:517:28) 182024-05-13T18:32:38.375Z at emitErrorNT (node:internal/streams/destroy:151:8) 192024-05-13T18:32:38.376Z at emitErrorCloseNT (node:internal/streams/destroy:116:3) 202024-05-13T18:32:38.378Z at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { 212024-05-13T18:32:38.380Z errno: -3, 222024-05-13T18:32:38.382Z code: 'Z_DATA_ERROR' 232024-05-13T18:32:38.383Z } 242024-05-13T18:32:38.386Z 252024-05-13T18:32:38.388Z Node.js v18.19.1
Hello and thank you for your interest in this Actor!
Our web-scraping library Crawlee is indeed having some issues with processing the sitemap on this domain. I already created a GitHub issue for this (see here) and our team will look into this soon.
In the meantime, you can simply disable the sitemap discovery (the Consider URLs from Sitemaps) option for this run. This way, the Actor won't try to access the sitemap and won't fail. Keep in mind that the sitemap discovery is only a supportive mechanism (and in most cases, you should get the same results in the same amount of time with and without it).
I'll keep you posted with any updates regarding this issue. Thank you! (and sorry for the inconvenience.)
Hello again! Just letting you know that this issue has been fixed in the latest release. This Actor should be able to process sitemaps correctly regardless of the compression.
I'll close this issue now, but feel free to reopen it if the issue resurfaces (although it shouldn't :)). Thank you again for your patience!
apricot_orange
I don't see how we can reopen the issue via the web so I'm responding here. I'm not sure if it's related but we had a sitemap based crawl today that didn't seem to be making any progress:
https://console.apify.com/organization/5WhuE8XiPsnLiYsmv/actors/runs/0s7yHkSxax9vQOcIO#log
Hello - this is a known issue (https://console.apify.com/actors/aYG0l9s7dbB7j3gbS/issues/5AfOIAxLtcJYZnDNy) and we're currently looking into the fix.
Right now, the Actor needs to parse the entire sitemap before processing the requests. If the sitemap is large enough, this takes a lot of time and can look like the Actor is stuck. We're implementing a non-blocking sitemap parser (PR here) that would allow us to start processing the requests during the sitemap parsing (concurrently).
We'll keep you updated once we make some progress with this (hopefully this / next week). Thank you for your patience!
Apify- 2.8k monthly users
- 317 stars
- 100.0% runs succeeded
- 4 days response time
- Created in Mar 2023
- Modified 1 day ago
