
Website Content Crawler
Pricing
Pay per usage
Go to Store


Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
3.6 (39)
Pricing
Pay per usage
1413
Total users
54K
Monthly users
7.9K
Runs succeeded
>99%
Issues response
7.6 days
Last modified
4 days ago
how to Crawl behind login?
Closed
Visualife opened this issue
10 months ago
how to Crawl behind login? thank you

10 months ago
Hello and thank you for your interest in Website Content Crawler! The most straightforward way to do this is to use the "Initial cookies" input parameter - you do the login locally and then carry over the cookies to the crawler.