No credit card required
Extended GPT Scraper
No credit card required
Extract data from any website and feed it into GPT via the OpenAI API. Use ChatGPT to proofread content, analyze sentiment, summarize reviews, extract contact details, and much more.
Rate limit needs to be handled.
Closed
I've been bumping up against the rate limit, unfortunately. Also, it seems OpenAI isn't currently accepting requests for an increased GPT-4 rate limit and my queries don't work well with GPT3.5, so I'm a bit stuck at the moment.
When I try to beef up the resources for the scraper, it quickly hits the rate limit and just keeps on trying, until it finally gives up and moves on to the next record. The process just keeps repeating, and I'm not getting any valuable output.
I tried another approach of cutting down to a machine spec until I don't get the rate-limit and it isn't until 512MB RAM, but then I ended up with memory limitations and the request just crashes. So, that's another dead end.
My suggestion on how to resolve this is:
Read and use the rate-limit values provided in the header reply from OpenAI. If the header indicates "x-ratelimit-limit-requests X", pace the requests to X/min.
How about we add a rate limit with TPM and RPM in the input menu? This could help us stay under the limit, and the user might want to have some control and limit the rate lower than the maximum allowed for a variety of reasons.
Lastly, when a rate limit response comes back, let's take a break from making any new requests for the duration of 'x-ratelimit-reset-requests' and 'x-ratelimit-reset-tokens'. If the rate limit issues persist, try an exponential backoff strategy until things look better.
designer_filter
same issue here
Hey Markus,
feature of new input fields makes sense and I consider to adding these in some way.
I will let you know when it will be done.
Hi, little update, it's in our backlog, unfortunately not a priority at the moment. Nevertheless, some rate limit management is a good idea, we hope to get back to it in the foreseeable future. Thanks for the suggestion and your patience!
Hi again, I´m going to close the issue for now, but we´re still hoping to get back to it. It might take some time though. Happy Scraping in the meantime!
Jakub Drobník- 74 monthly users
- 95.2% runs succeeded
- 1.9 days response time
- Created in Jun 2023
- Modified 1 minute ago
