Website Content Crawler
Pricing
Pay per usage
Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
4.6 (38)
Pricing
Pay per usage
1.1k
Monthly users
6k
Runs succeeded
>99%
Response time
2.3 days
Last modified
7 days ago
Crawling fails when URLs respond with application/pdf
I have a number of use cases where links go to PDF documents - which should use the raw HTML downloader - but where the URLs do not end in .pdf. In the linked example run, the URLs end in "pdf", but don't have an apparent file extension. When this happens, the crawler fails - it looks like it's attempting to use the browser-based crawl, and rejecting these pages.
Ideally, the decision of which crawler to use would be based on the MIME type of the response after fetching. Failing that - is there any way to configure different heuristics for how to handle these URLs?
Hi,
This behavior is expected when the crawler downloads a file (it throws an exception for that). If you want to access the file, you need to set "saveFiles": true or (in UI "Output settings" -> "Save files") and the files will be accessible in Key Value storage. Please see this run:
https://console.apify.com/view/runs/QLJaWGZRHSb20u0X3
Let me know if this does not work for you or if you have any questions.
Jakub
Pricing
Pricing model
Pay per usageThis Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.
