Twitter followers scraper avatar
Twitter followers scraper
Try for free

3 days trial then $40.00/month - No credit card required now

View all Actors
Twitter followers scraper

Twitter followers scraper

curious_coder/twitter-scraper
Try for free

3 days trial then $40.00/month - No credit card required now

Scrape followers, following and subscriptions list of any twitter profile

User avatar

Need advice.

Closed

extraordinary_muffin opened this issue
6 months ago

If I have to scrape 20M followers each of 20 handles (each handle having 20M followers) then which plan to take? How many can be scraped per week and per month? Can i run 20 concurrent threads of the actor in the same plan? Please guide so that it takes minimum possible time to scrape.

User avatar

You can't scrape too fast or simultaneously as this scraper uses cookie of your twitter account and twitter can block you for that. But you can run multiple instances with different accounts cookies and proxies to reduce time required to scrape. Other questions i don't have exact numbers or answers. You have to find out

User avatar

extraordinary_muffin

6 months ago

Thanks!

User avatar

extraordinary_muffin

6 months ago
  1. Suppose X blocks the account/cookie session in 1-2 weeks. @100K scrapes per day, there will be 1M scrapes before it blocks. Is 1M safe "minimum" figure to consider for planning scraping schedule, by your practical experience?

  2. After blocking and getting a store id with 1M scrapes from above run, when re-run is initiated with another account/cookie parameters, will the newly incorporated feature of inputting store id will begin with 0 or the 1st scrape after the 1Mth scrape i.e. the new csv will have count of 1 or 1000001?

  3. If the new csv count is 1000001, will there be repeats i.e. will 1000001 onwards scraped data will be same as previously (0-1000000) scraped data or the system will auto check for duplicate scraping or the system will auto check for duplicate before generating new csv?

Please guide on above as there will be new cookies post blocking and i don't want duplicate data.

Developer
Maintained by Community
Actor metrics
  • 120 monthly users
  • 88.6% runs succeeded
  • 6.4 days response time
  • Created in Jun 2023
  • Modified 21 days ago