Extended GPT Scraper avatar
Extended GPT Scraper
Try for free

No credit card required

View all Actors
Extended GPT Scraper

Extended GPT Scraper

drobnikj/extended-gpt-scraper
Try for free

No credit card required

Extract data from any website and feed it into GPT via the OpenAI API. Use ChatGPT to proofread content, analyze sentiment, summarize reviews, extract contact details, and much more.

User avatar

Rate limit needs to be handled.

Open

maroon_herb opened this issue
9 months ago

I've been bumping up against the rate limit, unfortunately. Also, it seems OpenAI isn't currently accepting requests for an increased GPT-4 rate limit and my queries don't work well with GPT3.5, so I'm a bit stuck at the moment.

When I try to beef up the resources for the scraper, it quickly hits the rate limit and just keeps on trying, until it finally gives up and moves on to the next record. The process just keeps repeating, and I'm not getting any valuable output.

I tried another approach of cutting down to a machine spec until I don't get the rate-limit and it isn't until 512MB RAM, but then I ended up with memory limitations and the request just crashes. So, that's another dead end.

My suggestion on how to resolve this is:

Read and use the rate-limit values provided in the header reply from OpenAI. If the header indicates "x-ratelimit-limit-requests X", pace the requests to X/min.

How about we add a rate limit with TPM and RPM in the input menu? This could help us stay under the limit, and the user might want to have some control and limit the rate lower than the maximum allowed for a variety of reasons.

Lastly, when a rate limit response comes back, let's take a break from making any new requests for the duration of 'x-ratelimit-reset-requests' and 'x-ratelimit-reset-tokens'. If the rate limit issues persist, try an exponential backoff strategy until things look better.

User avatar

designer_filter

9 months ago

same issue here

User avatar

Hey Markus,

feature of new input fields makes sense and I consider to adding these in some way.

I will let you know when it will be done.

User avatar

Hi, little update, it's in our backlog, unfortunately not a priority at the moment. Nevertheless, some rate limit management is a good idea, we hope to get back to it in the foreseeable future. Thanks for the suggestion and your patience!

Developer
Maintained by Apify
Actor metrics
  • 80 monthly users
  • 95.0% runs succeeded
  • 2.7 days response time
  • Created in Jun 2023
  • Modified 17 days ago