
Website Content Crawler
Pricing
Pay per usage

Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
4.6 (38)
Pricing
Pay per usage
1310
Total users
49.4k
Monthly users
6.9k
Runs succeeded
>99%
Issue response
3.8 days
Last modified
7 days ago
Issue Crawling Content from Paid Websites Like New York Times
Open
Tried crawling articles from the New York Times, but it didn’t work. No articles were scraped—only some unrelated page details were retrieved.

Hi, thank you for trying Website Content Crawler (WCC)! Some pages may require tinkering since it's a generic solution. Try a specific Actor like theo/new-york-times-scraper or timgreen/fast-news-scraper. Let me know if these work!