Twitter Scraper

What data can Twitter Scraper extract?

Twitter Scraper crawls specified Twitter profiles and URLs, and extracts:

🔍 User information, such as name, Twitter handle (username), location, follower/following count, profile URL/image/banner, date of creation.

🐦 List of tweets, retweets, and replies from profiles.

📊 Statistics for each tweet: favorites, replies, and retweets for each tweet.

Our Twitter Scraper enables you to extract large amounts of data from Twitter. It lets you do much more than the Twitter API, because it doesn't have rate limits and you don't even need to have a Twitter account, a registered app, or Twitter API key.

You can crawl based on a list of Twitter handles or just by using a Twitter post URL.

Why use Twitter Scraper?

Scraping Twitter will give you access to more than 500 million tweets posted every day. You can use that data in lots of different ways:

🔍 Track discussions about your brand, products, country, or city.

👥 Monitor your competitors and see how popular they really are, and how you can get a competitive edge.

🔎 Keep an eye on new trends, attitudes, and fashions as they emerge.

🧠 Use the data to train AI models or for academic research.

😊 Track sentiment to make sure your investments are protected.

🚫 Fight fake news by understanding the pattern of how misinformation spreads.

✈️ Explore discussions about travel destinations, services, amenities, and take advantage of local knowledge.

💰 Analyze consumer habits and develop new products or target underdeveloped niches.

How do I use Twitter Scraper?

It’s quite easy: pick Twitter Scraper, choose a profile, hashtag or tweet to scrape, and click the Start button. If you need guidance on how to run the scraper, you can read our step-by-step tutorial or watch a short video tutorial ▷ on YouTube.

How many results can you scrape with Twitter scraper?

Twitter scraper can return up to 3000 results on average. However, you have to keep in mind that scraping twitter.com has many variables to it and may cause the results to fluctuate case by case. There’s no one-size-fits-all-use-cases number. The maximum number of results may vary depending on the complexity of the input, location, and other factors. Some of the most frequent cases are:

• website gives a different number of results depending on the type/value of the input
• website has an internal limit that no scraper can cross
• scraper has a limit that we are working on improving

Therefore, while we regularly run Actor tests to keep the benchmarks in check, the results may also fluctuate without our knowing. The best way to know for sure for your particular use case is to do a test run yourself.

How much will scraping Twitter cost you?

When it comes to scraping, it can be challenging to estimate the resources needed to extract data as use cases may vary significantly. That's why the best course of action is to run a test scrape with a small sample of input data and limited output. You’ll get your price per scrape, which you’ll then multiply by the number of scrapes you intend to do.

Watch this video for a few helpful tips. And don't forget that choosing a higher plan will save you money in the long run.

Is it legal to scrape Twitter?

It is legal to scrape Twitter to extract publicly available information, but you should be aware that the data extracted might contain personal data. Personal data is protected by GDPR in the European Union and by other regulations around the world. You should not scrape personal data unless you have a legitimate reason to do so. If you're unsure whether your reason is legitimate, consult your lawyers. You can also read our blog post on the legality of web scraping.

Tips and tricks

Using the URL option

The default option is to scrape using Twitter handles (usernames), but you can also scrape by Twitter URLs. If you want to use the URL option, these are the supported Twitter URL types ⬇️

• Profiles: https://twitter.com/elonmusk
• Statuses: https://twitter.com/elonmusk/status/1356381230925635591
• Topics: https://twitter.com/i/topics/933033311844286464
• Events: https://twitter.com/i/events/1354736314923372544
• Explore: https://twitter.com/explore
• List: https://twitter.com/i/lists/1611381299687694336

Important notice

• The Actor only extracts publicly available information.
• As Twitter hid its search functionality behind login in April 2023, this functionality no longer works on this Actor either. Try different ones you can find in Apify Store.
• You can still scrape profile names and tweet URLs though.

Input

Twitter Scraper has the following input options:

Twitter data output

The results will be wrapped into a dataset which you can find in the Storage tab. Here's an excerpt from the dataset you'd get if you apply the input parameters above:

You can download the resulting dataset in various formats such as JSON, HTML, CSV or Excel. Each item in the dataset will contain a separate tweet following this format:

{
   "user":{
      "protected":false,
      "created_at":"2009-06-02T20:12:29.000Z",
      "default_profile_image":false,
      "description":"",
      "fast_followers_count":0,
      "favourites_count":19158,
      "followers_count":130769125,
      "friends_count":183,
      "has_custom_timelines":true,
      "is_translator":false,
      "listed_count":117751,
      "location":"",
      "media_count":1435,
      "name":"Elon Musk",
      "normal_followers_count":130769125,
      "possibly_sensitive":false,
      "profile_banner_url":"<https://pbs.twimg.com/profile_banners/44196397/1576183471>",
      "profile_image_url_https":"<https://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg>",
      "screen_name":"elonmusk",
      "statuses_count":23422,
      "translator_type":"none",
      "verified":true,
      "withheld_in_countries":[
         
      ],
      "id_str":"44196397"
   },
   "id":"1633026246937546752",
   "conversation_id":"1632363525405392896",
   "full_text":"@MarkChangizi Sweden’s steadfastness was incredible!",
   "reply_count":243,
   "retweet_count":170,
   "favorite_count":1828,
   "hashtags":[
      
   ],
   "symbols":[
      
   ],
   "user_mentions":[
      {
         "id_str":"49445813",
         "name":"Mark Changizi",
         "screen_name":"MarkChangizi"
      }
   ],
   "urls":[
      
   ],
   "media":[
      
   ],
   "url":"<https://twitter.com/elonmusk/status/1633026246937546752>",
   "created_at":"2023-03-07T08:46:12.000Z",
   "is_quote_tweet":false,
    "replying_to_tweet":"<https://twitter.com/MarkChangizi/status/1632363525405392896>",
   "startUrl":"<https://twitter.com/elonmusk/with_replies>"
}

Want more tools for scraping Twitter?

🔎 Tweet Flash - Twitter Scraper	🔍 Twitter URL Scraper
🎥 Twitter Video Downloader	🪣 Twitter List Scraper
🛰️ Twitter Spaces Scraper	🧭 Twitter Explorer

Not your cup of tea? Build your own scraper

Twitter scraper doesn’t exactly do what you need? You can always build your own! We have various scraper templates in Python, JavaScript, and TypeScript to get you started. Alternatively, you can write it from scratch using our open-source library Crawlee. You can keep the scraper to yourself or make it public by adding it to Apify Store (and find users for it).

Or let us know if you need a custom scraping solution.

Integrations and Twitter Scraper

Last but not least, Twitter Scraper can be connected with almost any cloud service or web app thanks to integrations on the Apify platform. You can integrate with Make, Zapier, Slack, Airbyte, GitHub, Google Sheets, Google Drive, and more. Or you can use webhooks to carry out an action whenever an event occurs, e.g. get a notification whenever Twitter Scraper successfully finishes a run.

Using Twitter Scraper with the Apify API

The Apify API gives you programmatic access to the Apify platform. The API is organized around RESTful HTTP endpoints that enable you to manage, schedule, and run Apify actors. The API also lets you access any datasets, monitor actor performance, fetch results, create and update versions, and more.

To access the API using Node.js, use the apify-client NPM package. To access the API using Python, use the apify-client PyPI package.

Check out the Apify API reference docs for full details or click on the API tab for code examples.

Your feedback

We’re always working on improving the performance of our Actors. So if you’ve got any technical feedback for Twitter Scraper or simply found a bug, please create an issue on the Actor’s Issues tab in Apify Console.