Website Content Crawler

Thank you for your interest in the website content crawler.

However, I'm not entirely sure what you are trying to achieve?

Are you are referring to a ChatGPT Plugin or Action (this guide)? Please note that the Website Content Crawler is not the best choice in this case due to the 45-second timeout on plugins/actions calls. Website content crawler is used to crawl entire website and most probably will take longer than that. You might want to check Web Scraper Actor, which might be faster

If you are referring to (having problems with) the OpenAI Vector Store integration I can help you to set up that too.

Thanks! I will switch to Web Scraper Actor then.

But my main problem it the integration with OpenAI. So please help me with that. Thanks

This is what happens, see screen shots. thanks

I tried to replicate it step by step using this guide and it worked.

Additionally, I tested the website content crawler and it worked. Please see the testing screenshot and chat screenshot for reference.

For the website content crawler, when copying the OpenAPI specification, make sure to locate the settings maxCrawlPages and set the default value to 1. This way, the crawler will scrape only a single page and should complete the task within the 45-second timeout enforced by ChatGPT on actions.

1"maxCrawlPages": {
2            "title": "Max pages",
3            "minimum": 0,
4            "type": "integer",
5            "description": "The maximum number pages to crawl. It includes the start URLs, pagination pages, pages with no content, etc. The crawler will automatically finish after reaching this number. This setting is useful to prevent accidental crawler runaway.",
6            "default": 1
7          },

In the screenshots you have the Custom GPT instructions, it should work.

Regarding the OpenAI Vector Store, have you read this article? There are some steps how to setup OpenAI Vector Store.

If it doesn't help, I can create a more detailed guide.