Website Content Crawler avatar
Website Content Crawler

Pricing

Pay per usage

Go to Store
Website Content Crawler

Website Content Crawler

Developed by

Apify

Apify

Maintained by Apify

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

3.6 (39)

Pricing

Pay per usage

1413

Total users

54K

Monthly users

7.9K

Runs succeeded

>99%

Issues response

7.6 days

Last modified

4 days ago

VI

how to Crawl behind login?

Closed

Visualife opened this issue
10 months ago

how to Crawl behind login? thank you

janbuchar avatar

Hello and thank you for your interest in Website Content Crawler! The most straightforward way to do this is to use the "Initial cookies" input parameter - you do the login locally and then carry over the cookies to the crawler.