Website Content Crawler avatar
Website Content Crawler

Pricing

Pay per usage

Go to Store
Website Content Crawler

Website Content Crawler

Developed by

Apify

Apify

Maintained by Apify

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

4.2 (40)

Pricing

Pay per usage

1398

Total users

54K

Monthly users

8K

Runs succeeded

>99%

Issues response

6.8 days

Last modified

5 days ago

TI

Adsterra .com

Open

Tijjeboy opened this issue
2 months ago

How to remove anonymous proxy

TI

Tijjeboy

2 months ago

How to remove anonymous proxy

jiri.spilka avatar

Hi, thank you for trying Website Content Crawler.

It looks like there’s an issue in the UI, currently, the "No proxy" option can’t be selected. I’ll report it internally.

That said, why do you need to disable the proxy?
In most cases, you don’t want to do that. Without a proxy, the crawler is likely to get blocked.
Using a proxy is strongly recommended, datacenter proxies are fast and inexpensive.

Thank you, Jiri