Twitter Explorer
7 days trial then $30.00/month - No credit card required now
Twitter Explorer
7 days trial then $30.00/month - No credit card required now
🐦🐦⬛🐤 All-in-one solution to scrape every aspect of Twitter.com
<elonmusk> error #88: Rate limit exceeded.
Closed
Link to my run: https://console.apify.com/view/runs/QcJtPTypaQeaz4C55
Hi Jupri, I received the error #88 when I executed the run above. Does the error come from the Apify end or from the Twitter end? Is there any thing I can do to resolve it?
Context: I wanted to retrieve all tweets from a list of selected accounts within the date range 2022-01-01 through 2023-06-30. To get all of their tweets, I set the "limit" parameter as large as possible (i.e., 1M). If there is any better way to handle my request, please feel free to let me know. Thank you!
Hi, sorry for the inconvenience. Yep. the error comes from twitter. There`s nothing we can do about it except wait for several hour, and let it recover. (lets pray elonmusk will remove this limit rate soon). Or, you could try using your own token (auth_token) for a while.
FYI: rate limit for SEARCH is 50 requests per 15 minutes every request give you 20 posts (maybe less), so total about 1_000 results per 15 minutes. you could throttle (slowdown) your requests according to this boundary
yiting_shen_001
Hi Cat, thanks for your reply! I have two follow-up questions:
- Is ~1,000 results per 15 minutes a hard cap? I once managed to retrieve ~1,500 results within 10 minutes using Jupri. Was this because Jupri was doing some optimization in the background (e.g., distributing the tasks to different workers)?
- Would setting time.sleep()'s in my code work out in this situation? What do you think is the best way to set time.sleep()'s between Jupri requests to get all tweets from a list of selected accounts within the date range 2022-01-01 through 2023-06-30? Thanks so much for your advice in advance!
Using sleep in your end could help.
If my calculation correct:
50 requests per 15 minutes = 1 request per 18 seconds (after 1 request, pause for 18 seconds max)
It depend on number of results you received, for example:
on first request you get 100 results: 100/20 results = 5 requests, therefore you need to pause 5x18 = 90 seconds max (more exact = 90 minus seconds the request took)
Illustration:
00:00:00 first request ...
00:00:05 100 results received. then SLEEP for 90-5 = 85 seconds
00:01:30 next request ( ... so on)
yiting_shen_001
Hi Cat, thanks for your reply. Regarding the limit issue, may I ask if there is a hard cap for the number of tweets to retrieve within one month?
cat- 11 monthly users
- 3 stars
- 86.3% runs succeeded
- 9.9 hours response time
- Created in May 2023
- Modified 4 months ago
