![Website Content Crawler avatar](https://images.apifyusercontent.com/hchWWpDJQyd-Nh_2beH12X2c6MBfXEQc59HAA24sOAY/rs:fill:250:250/aHR0cHM6Ly9hcGlmeS1pbWFnZS11cGxvYWRzLXByb2QuczMuYW1hem9uYXdzLmNvbS9hWUcwbDlzN2RiQjdqM2diUy9QZlRvRU5rSlp4YWh6UER1My1DbGVhblNob3RfMjAyMy0wMy0yOF9hdF8xMC40MC4yMF8yeC5wbmc.webp)
No credit card required
![Website Content Crawler](https://images.apifyusercontent.com/hchWWpDJQyd-Nh_2beH12X2c6MBfXEQc59HAA24sOAY/rs:fill:250:250/aHR0cHM6Ly9hcGlmeS1pbWFnZS11cGxvYWRzLXByb2QuczMuYW1hem9uYXdzLmNvbS9hWUcwbDlzN2RiQjdqM2diUy9QZlRvRU5rSlp4YWh6UER1My1DbGVhblNob3RfMjAyMy0wMy0yOF9hdF8xMC40MC4yMF8yeC5wbmc.webp)
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.
Do you want to learn more about this Actor?
Get a demoan issue with the apify_website-content-crawler
Closed
I followed one of your tutorials to integrate my GPT with your actor, but upon testing the actor, chatgpt returns: I'll proceed with these parameters now.
[debug] Calling HTTP endpoint [debug] Response received Error talking to
It seems there's an issue with the apify_website-content-crawler operation, as it's not being recognized.
I recommend reaching out to Apify support for further assistance or checking the API documentation for any changes. If there's anything specific you'd like to try next or another task you'd like to perform, please let me know! What Should I do?
![jiri.spilka avatar](https://apify-image-uploads-prod.s3.us-east-1.amazonaws.com/vaTaQJYTC5hi1UCDf/Rjpb1gcqnqhohvfwc-spilka_2024_apify.jpg)
Thank you for your interest in the website content crawler.
However, I'm not entirely sure what you are trying to achieve?
Are you are referring to a ChatGPT Plugin or Action (this guide)? Please note that the Website Content Crawler is not the best choice in this case due to the 45-second timeout on plugins/actions calls. Website content crawler is used to crawl entire website and most probably will take longer than that. You might want to check Web Scraper Actor, which might be faster
If you are referring to (having problems with) the OpenAI Vector Store integration I can help you to set up that too.
Chris_the
Thanks! I will switch to Web Scraper Actor then.
But my main problem it the integration with OpenAI. So please help me with that. Thanks
Chris_the
This is what happens, see screen shots. thanks
![jiri.spilka avatar](https://apify-image-uploads-prod.s3.us-east-1.amazonaws.com/vaTaQJYTC5hi1UCDf/Rjpb1gcqnqhohvfwc-spilka_2024_apify.jpg)
I tried to replicate it step by step using this guide and it worked.
Additionally, I tested the website content crawler and it worked. Please see the testing screenshot and chat screenshot for reference.
For the website content crawler, when copying the OpenAPI specification, make sure to locate the settings maxCrawlPages
and set the default value to 1. This way, the crawler will scrape only a single page and should complete the task within the 45-second timeout enforced by ChatGPT on actions.
1"maxCrawlPages": { 2 "title": "Max pages", 3 "minimum": 0, 4 "type": "integer", 5 "description": "The maximum number pages to crawl. It includes the start URLs, pagination pages, pages with no content, etc. The crawler will automatically finish after reaching this number. This setting is useful to prevent accidental crawler runaway.", 6 "default": 1 7 },
In the screenshots you have the Custom GPT instructions, it should work.
Regarding the OpenAI Vector Store, have you read this article? There are some steps how to setup OpenAI Vector Store.
If it doesn't help, I can create a more detailed guide.
![jiri.spilka avatar](https://apify-image-uploads-prod.s3.us-east-1.amazonaws.com/vaTaQJYTC5hi1UCDf/Rjpb1gcqnqhohvfwc-spilka_2024_apify.jpg)
I'm closing this issue as it is more related to GPT Actions rather than the Website Content Crawler.
If you encounter any issues with the OpenAI Vector Store, please feel free to open a new issue there
- 2.8k monthly users
- 434 stars
- 99.9% runs succeeded
- 2.9 days response time
- Created in Mar 2023
- Modified 3 days ago