Twitter URL Scraper

ℹ️ This Twitter scraper only collects data that’s publicly available. This means data that’s accessible without logging in to Twitter and without accepting Twitter’s terms of use. Please note that if you accepted Twitter’s terms of use, your ability to scrape Twitter data may be limited. If that’s the case, please review the terms and make an informed decision yourself.

What does Twitter URL scraper do?

Twitter URL scraper will enable you to get data from Twitter.com.

Twitter URL scraper can scrape:

• User information, such as name, Twitter handle (username), location, follower/following count, profile URL/image/banner, date of creation
• List of tweets, retweets, and replies from profiles
• Statistics for each tweet: favorites, replies, and retweets for each tweet

Why scrape Twitter.com?

Twitter.com has 217 million active daily users and is a great source of data for marketing and research based on social media.

Here are just some of the ways you could use that data:

• Track discussions about your brand, products, country, or city.
• Monitor your competitors and their popularity, and get a competitive edge.
• Keep an eye on new trends and fashions as they emerge.
• Use the data to train AI models or for academic research.
• Track sentiment to make sure your investments are protected.
• Fight fake news by identifying the spreading pattern of misinformation.
• Explore discussions about travel destinations, services, amenities, and take advantage of local knowledge.
• Analyze consumer habits and develop new products or target underdeveloped niches.

How to scrape Twitter.com

It's easy to scrape Twitter.com with Twitter URL scraper. Just follow these few steps and you'll get your data in a few minutes.

1. Click on Try for free.
2. Copy a URL from Twitter.com which you want to scrape and paste it to the actor.
3. Click on Start.
4. When Twitter URL scraper has finished, preview or download your data from the Dataset tab.

Supported Twitter URL types

• Profiles: https://twitter.com/elonmusk
• Statuses: https://twitter.com/elonmusk/status/1356381230925635591

How many results can you scrape with Twitter URL scraper?

Twitter URL scraper can return up to 800 results per URL on average (or of course 1 result if it is a status url). However, you have to keep in mind that scraping twitter.com has many variables to it and may cause the results to fluctuate case by case. There’s no one-size-fits-all-use-cases number. The maximum number of results may vary depending on the complexity of the input, location, and other factors. Some of the most frequent cases are:

• website gives a different number of results depending on the type/value of the input
• website has an internal limit that no scraper can cross
• scraper has a limit that we are working on improving

Therefore, while we regularly run Actor tests to keep the benchmarks in check, the results may also fluctuate without our knowing. The best way to know for sure for your particular use case is to do a test run yourself.

Results

The actor stores its results into the default dataset associated with the actor run. The data can be downloaded in machine-readable formats such as JSON, HTML, CSV or Excel.

Each item in the dataset will contain a separate tweet that follows this format:

{
  "user": {
    "id_str": "44196397",
    "name": "Elon Musk",
    "screen_name": "elonmusk",
    "location": "",
    "description": "",
    "followers_count": 42583621,
    "fast_followers_count": 0,
    "normal_followers_count": 42583621,
    "friends_count": 104,
    "listed_count": 59150,
    "created_at": "2009-06-02T20:12:29.000Z",
    "favourites_count": 7840,
    "verified": true,
    "statuses_count": 13360,
    "media_count": 801,
    "profile_image_url_https": "https://pbs.twimg.com/profile_images/1295975423654977537/dHw9JcrK_normal.jpg",
    "profile_banner_url": "https://pbs.twimg.com/profile_banners/44196397/1576183471",
    "has_custom_timelines": true,
    "advertiser_account_type": "promotable_user",
    "business_profile_state": "none",
    "translator_type": "none"
  },
  "id": "1338857124508684289",
  "conversation_id": "1338390123373801472",
  "full_text": "@CyberpunkGame The objective reality is that it is impossible to run an advanced game well on old hardware. This is a much more serious issue: https://t.co/OMNCTa9hJY",
  "reply_count": 792,
  "retweet_count": 669,
  "favorite_count": 17739,
  "hashtags": [],
  "symbols": [],
  "user_mentions": [
    {
      "screen_name": "CyberpunkGame",
      "name": "Cyberpunk 2077",
      "id_str": "821102114"
    }
  ],
  "urls": [
    {
      "url": "https://t.co/OMNCTa9hJY",
      "expanded_url": "https://www.pcgamer.com/the-more-time-i-spend-in-cyberpunk-2077s-world-the-less-i-believe-in-it/",
      "display_url": "pcgamer.com/the-more-time-…"
    }
  ],
  "url": "https://twitter.com/elonmusk/status/1338857124508684289",
  "created_at": "2020-12-15T14:43:07.000Z"
}

Integrations and Twitter URL Scraper

Last but not least, Twitter URL Scraper can be connected with almost any cloud service or web app thanks to integrations on the Apify platform. You can integrate with Make, Zapier, Slack, Airbyte, GitHub, Google Sheets, Google Drive, and more. Or you can use webhooks to carry out an action whenever an event occurs, e.g. get a notification whenever Twitter URL Scraper successfully finishes a run.

Using Twitter URL Scraper with the Apify API

The Apify API gives you programmatic access to the Apify platform. The API is organized around RESTful HTTP endpoints that enable you to manage, schedule, and run Apify actors. The API also lets you access any datasets, monitor actor performance, fetch results, create and update versions, and more.

To access the API using Node.js, use the apify-client NPM package. To access the API using Python, use the apify-client PyPI package.

Check out the Apify API reference docs for full details or click on the API tab for code examples.

Is it legal to scrape Twitter.com?

Note that personal data is protected by GDPR in the European Union and by other regulations around the world. You should not scrape personal data unless you have a legitimate reason to do so. If you're unsure whether your reason is legitimate, consult your lawyers. We also recommend that you read our blog post: is web scraping legal?.

Need a more advanced Twitter scraper?

If you need more detailed data from Twitter or need to use different input, try our Twitter Scraper. It's a little more complicated to set up, but you can customize it as you want.

You can read our step-by-step tutorial on how to scrape Twitter if you need some guidance on how to run the scraper.

Want more tools for scraping Twitter?

Not your cup of tea? Build your own scraper

Twitter URL scraper doesn’t exactly do what you need? You can always build your own! We have various scraper templates in Python, JavaScript, and TypeScript to get you started. Alternatively, you can write it from scratch using our open-source library Crawlee. You can keep the scraper to yourself or make it public by adding it to Apify Store (and find users for it).

Your feedback

We’re always working on improving the performance of our Actors. So if you’ve got any technical feedback for Twitter URL scraper or simply found a bug, please create an issue on the Actor’s Issues tab.