Website Content Crawler avatar
Website Content Crawler

Pricing

Pay per usage

Go to Store
Website Content Crawler

Website Content Crawler

Developed by

Apify

Apify

Maintained by Apify

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

4.6 (38)

Pricing

Pay per usage

1253

Total users

47k

Monthly users

6.6k

Runs succeeded

>99%

Response time

4.7 days

Last modified

2 days ago

IP

Unable to crawl https://openai.com/index/extracting-concepts-from-gpt-4/

Closed

imda_peckyoke opened this issue
8 months ago

I get the following text when crawling https://openai.com/index/extracting-concepts-from-gpt-4/. The extracted text: Application error: a client-side exception has occurred (see the browser console for more information)..

Seems other articles on this site is able to crawl. Just this particular articles causes issue.

jindrich.bar avatar

Hello, and thank you for your interest in this Actor

It indeed seems that the Actor has some problems loading this website. This is likely connected to the Actor using Mozilla Firefox, which doesn't have the CookieStore API available. What's curious is that I can load the page in Firefox locally without problems.

You can solve the issue on this specific website by switching to the Cheerio crawler type. Since there is no essential client-side JavaScript, the Cheerio crawler can extract all the data from the webpage (see my example run).

I'll keep this issue open to track the underlying problem. Feel free to ask additional questions if you have any. Cheers!

dusan.vystrcil avatar

Hi @imda_peckyoke, if you are still experiencing issues, please let us know. For now, I am closing this one.