Twitter Explorer avatar
Twitter Explorer

Pricing

$30.00/month + usage

Go to Store
Twitter Explorer

Twitter Explorer

jupri/twitter-scraper

Developed by

cat

Maintained by Community

🐦🐦‍⬛🐤 All-in-one solution to scrape every aspect of Twitter.com

5.0 (1)

Pricing

$30.00/month + usage

3

Monthly users

18

Runs succeeded

95%

Last modified

8 months ago

YS

<elonmusk> error #88: Rate limit exceeded.

Closed
yiting_shen_001 opened this issue
2 years ago

Link to my run: https://console.apify.com/view/runs/QcJtPTypaQeaz4C55

Hi Jupri, I received the error #88 when I executed the run above. Does the error come from the Apify end or from the Twitter end? Is there any thing I can do to resolve it?

Context: I wanted to retrieve all tweets from a list of selected accounts within the date range 2022-01-01 through 2023-06-30. To get all of their tweets, I set the "limit" parameter as large as possible (i.e., 1M). If there is any better way to handle my request, please feel free to let me know. Thank you!

Jupri avatar

cat (Jupri)

2 years ago

Hi, sorry for the inconvenience. Yep. the error comes from twitter. There`s nothing we can do about it except wait for several hour, and let it recover. (lets pray elonmusk will remove this limit rate soon). Or, you could try using your own token (auth_token) for a while.

FYI: rate limit for SEARCH is 50 requests per 15 minutes every request give you 20 posts (maybe less), so total about 1_000 results per 15 minutes. you could throttle (slowdown) your requests according to this boundary

YS

yiting_shen_001

2 years ago

Hi Cat, thanks for your reply! I have two follow-up questions:

  1. Is ~1,000 results per 15 minutes a hard cap? I once managed to retrieve ~1,500 results within 10 minutes using Jupri. Was this because Jupri was doing some optimization in the background (e.g., distributing the tasks to different workers)?
  2. Would setting time.sleep()'s in my code work out in this situation? What do you think is the best way to set time.sleep()'s between Jupri requests to get all tweets from a list of selected accounts within the date range 2022-01-01 through 2023-06-30? Thanks so much for your advice in advance!
Jupri avatar

cat (Jupri)

2 years ago

Using sleep in your end could help. If my calculation correct: 50 requests per 15 minutes = 1 request per 18 seconds (after 1 request, pause for 18 seconds max)

It depend on number of results you received, for example: on first request you get 100 results: 100/20 results = 5 requests, therefore you need to pause 5x18 = 90 seconds max (more exact = 90 minus seconds the request took)

Illustration:

00:00:00 first request ... 00:00:05 100 results received. then SLEEP for 90-5 = 85 seconds 00:01:30 next request ( ... so on)

YS

yiting_shen_001

2 years ago

Hi Cat, thanks for your reply. Regarding the limit issue, may I ask if there is a hard cap for the number of tweets to retrieve within one month?

Pricing

Pricing model

Rental 

To use this Actor, you have to pay a monthly rental fee to the developer. The rent is subtracted from your prepaid usage every month after the free trial period. You also pay for the Apify platform usage.

Free trial

7 days

Price

$30.00