Website Content Crawler avatar
Website Content Crawler

Pricing

Pay per usage

Go to Store
Website Content Crawler

Website Content Crawler

Developed by

Apify

Apify

Maintained by Apify

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

4.6 (38)

Pricing

Pay per usage

1310

Total users

49.4k

Monthly users

6.9k

Runs succeeded

>99%

Issue response

3.8 days

Last modified

7 days ago

ON

Issue Crawling Content from Paid Websites Like New York Times

Open

onlinereach opened this issue
a month ago

Tried crawling articles from the New York Times, but it didn’t work. No articles were scraped—only some unrelated page details were retrieved.

jakub.kopecky avatar

Hi, thank you for trying Website Content Crawler (WCC)! Some pages may require tinkering since it's a generic solution. Try a specific Actor like theo/new-york-times-scraper or timgreen/fast-news-scraper. Let me know if these work!