Website Content Crawler
Pricing
Pay per usage
Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
4.6 (38)
Pricing
Pay per usage
1.1k
Monthly users
5.9k
Runs succeeded
>99%
Response time
2.3 days
Last modified
7 days ago
Crawler accesses pages and loads data correctly, but status code is 404
Hello, something strange is happening with a run:
- The crawler (Playwright) accesses the pages correctly and reads in the resulting html etc. without issues, but the httpStatusCode returned from the server for the initial GET request is 404. I do not know why the website does it like this.
- Is there any other indicator that signifies "all content has been loaded correctly"? Background is that I check for the statusCode in my application and if it is 404/400/etc., it simply skips processing. This makes sense to me, as in this case the html returned is usually just the error page. Therefore, I am looking for a workaround for this weird case.
Thank you!
Hello, and thank you for your interest in this Actor.
Note that this is not an issue with the Actor but rather the website you're scraping. The server returns the responses with the 404 status code even when accessed regularly via a browser.
To answer your second question - the web standards are very malleable (and client browsers are very forgiving because of this). While 404 should technically mean that the content hasn't been found, it doesn't stop the server from returning data in the response body (which can be any data, including the actual expected response). Unfortunately, there aren't many other signs you can use (and you have virtually no guarantee that the server won't lie again). My decision would probably be to accept the 404 responses for this one domain and keep an eye out for other, more severe errors. Or, of course, you can always try to analyze the content you receive (i.e., search for "Not found" substrings, etc.).
It would be great if everybody on the Internet stuck to the standards, huh? :)
I'll close this issue now, but feel free to ask additional questions, if you have any. Cheers!
cirez_d
Alright, makes sense, thank you! I thought there may have been another indicator that I could use, as I am using Playwright to handle the requests.
Pricing
Pricing model
Pay per usageThis Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.
