Website Content Crawler
Pricing
Pay per usage
Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
3.7 (41)
Pricing
Pay per usage
1537
Total users
59K
Monthly users
7.9K
Runs succeeded
>99%
Issues response
7.8 days
Last modified
2 days ago
Avoid scraping pages when redirected
Closed
Currently, if the crawler attempts to scrape a page that ultimately redirects the user, the redirected page is scraped. Is it possible to opt-out of this behavior and avoid scraping pages that redirect the user, or any other non-200 status codes?
Hello,
Thank you for trying the Website Content Crawler! Yes, you can control which URLs are included in the crawl using the ‘includeUrlGlob’ input variable. For more details, please refer to our input definitions page: https://apify.com/apify/website-content-crawler/input-schema
Let me know if you need any further adjustments!
AnyTeam
Hi Oscar-- sorry, what we're looking to do is avoid scraping URLs with specific response codes. Currently, if a page 404s, 500s, etc, the content is still scraped and returned by the crawler. Can those situations be avoided without knowing ahead of time which URLs to exclude from the run?
Hello, unfortunately, that's not possible. As a workaround, after the run is finished, you can filter the content of the pages and remove all the pages that have error codes in the 'httpStatusCode' attribute.
I’ll go ahead and close this issue now, but please feel free to ask additional questions or raise a new issue.
