Twitter (X) Profiles and Tweets Scraper avatar

Twitter (X) Profiles and Tweets Scraper

Try for free

2 hours trial then $15.00/month - No credit card required now

Go to Store
Twitter (X) Profiles and Tweets Scraper

Twitter (X) Profiles and Tweets Scraper

memo23/apify-twitter-profile-scraper
Try for free

2 hours trial then $15.00/month - No credit card required now

This actor allows you to scrape Twitter/X profiles, extract tweets, key metrics, metadata and gather insights such as tweet content, engagement metrics (likes, retweets, replies), and author details. It is handy for social media analysis, marketing research, and tracking public figures or brands.

Twitter Profiles and Tweets Scraper

Overview

This powerful scraper allows you to extract comprehensive data from Twitter (X) profiles and their tweets. It's designed to work with various input formats, including full profile URLs, usernames, and even Instagram profile URLs.

Features

  • Scrape detailed profile information
  • Extract tweets with comprehensive metadata
  • Supports multiple input URLs, including Twitter handles and profile URLs.
  • Username-only input support
  • Configurable concurrency and retry mechanisms
  • Optional inclusion of posts/tweets
  • Proxy configuration for enhanced reliability
  • Extracts tweets, user information, engagement metrics, tweet sources, and more.

Input Configuration

  1. Start URLs:

    • Full profile URLs: https://x.com/elonmusk or https://twitter.com/elonmusk
    • Usernames: adrian_horning_
    • Instagram URLs (for cross-platform profiles): https://www.instagram.com/robertsyfert/
  2. Max Concurrency: Set maximum concurrent requests (default: 10)

  3. Min Concurrency: Set minimum concurrent requests (default: 1)

  4. Max Request Retries: Set maximum retry attempts for failed requests (default: 100)

Input Data

Here's an example of how to set up a Twitter profile scraping task:

1{
2    "startUrls": [
3        "https://x.com/elonmusk",
4        "elonmusk",
5        "https://www.instagram.com/robertsyfert/"
6    ],
7    "maxConcurrency": 10,
8    "minConcurrency": 1,
9    "maxRequestRetries": 100
10}

Output Structure

1{
2  "type": "tweet",
3  "id": "1519480761749016577",
4  "url": "https://x.com/elonmusk/status/1519480761749016577",
5  "twitterUrl": "https://twitter.com/elonmusk/status/1519480761749016577",
6  "text": "Next I'm buying Coca-Cola to put the cocaine back in",
7  "retweetCount": 620127,
8  "replyCount": 179263,
9  "likeCount": 4434571,
10  "quoteCount": 166686,
11  "createdAt": "Thu Apr 28 00:56:58 +0000 2022",
12  "lang": "en",
13  "quoteId": null,
14  "bookmarkCount": 21208,
15  "isReply": false,
16  "card": {},
17  "place": {},
18  "source": "<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>",
19  "author": {
20    "type": "user",
21    "userName": "elonmusk",
22    "url": "https://x.com/elonmusk",
23    "twitterUrl": "https://twitter.com/elonmusk",
24    "id": "44196397",
25    "name": "Elon Musk",
26    "isVerified": true,
27    "isBlueVerified": true,
28    "verifiedType": "none",
29    "hasNftAvatar": false,
30    "profilePicture": "https://pbs.twimg.com/profile_images/1845482317860450309/OrD0ovmf_normal.jpg",
31    "coverPicture": "https://pbs.twimg.com/profile_banners/44196397/1726163678",
32    "description": "Read @America to understand why I'm supporting Trump for President",
33    "location": "",
34    "followers": 201720551,
35    "following": 787,
36    "protected": false,
37    "status": "",
38    "canDm": false,
39    "canMediaTag": false,
40    "advertiserAccountType": "promotable_user",
41    "analyticsType": "enabled",
42    "createdAt": "Tue Jun 02 20:12:29 +0000 2009",
43    "entities": {
44      "description": {
45        "urls": []
46      },
47      "url": {
48        "urls": [
49          {
50            "display_url": "TheAmericaPAC.org",
51            "expanded_url": "http://TheAmericaPAC.org",
52            "url": "https://t.co/DjyKIO6ePx",
53            "indices": [
54              0,
55              23
56            ]
57          }
58        ]
59      }
60    },
61    "fastFollowersCount": 0,
62    "favouritesCount": 81232,
63    "geoEnabled": false,
64    "hasCustomTimelines": true,
65    "hasExtendedProfile": false,
66    "isTranslator": false,
67    "mediaCount": 2592,
68    "profileBackgroundColor": "C0DEED",
69    "statusesCount": 54347,
70    "translatorTypeEnum": "None",
71    "withheldInCountries": [],
72    "affiliatesHighlightedLabel": {
73      "label": {
74        "url": {
75          "url": "https://twitter.com/X",
76          "urlType": "DeepLink"
77        },
78        "badge": {
79          "url": "https://pbs.twimg.com/profile_images/1683899100922511378/5lY42eHs_bigger.jpg"
80        },
81        "description": "X",
82        "userLabelType": "BusinessLabel",
83        "userLabelDisplayType": "Badge"
84      }
85    }
86  },
87  "extendedEntities": {},
88  "isRetweet": false,
89  "isQuote": false,
90  "media": []
91}

Output Format

The scraper provides rich, structured data for both profiles and tweets. Here's a detailed breakdown of the output fields:

Tweet Object

FieldTypeDescription
typeStringAlways "tweet" for tweet objects
idStringUnique identifier of the tweet
urlStringFull URL to the tweet on X.com
twitterUrlStringFull URL to the tweet on Twitter.com
textStringContent of the tweet
retweetCountNumberCount of retweets
replyCountNumberCount of replies
likeCountNumberCount of likes
quoteCountNumberCount of quote tweets
createdAtStringTimestamp of tweet creation
langStringLanguage code of the tweet
quoteIdStringnull
bookmarkCountNumberCount of bookmarks
isReplyBooleanIndicates if the tweet is a reply
cardObjectAdditional card information (e.g., for links)
placeObjectLocation information, if available
sourceStringApplication used to post the tweet
isRetweetBooleanIndicates if the tweet is a retweet
isQuoteBooleanIndicates if the tweet is a quote tweet
mediaArrayContains media objects (images, videos)
extendedEntitiesObjectAdditional entities (e.g., extended media info)

Author Object (User Profile)

FieldTypeDescription
typeStringAlways "user" for profile objects
userNameStringTwitter handle of the user
urlStringProfile URL on X.com
twitterUrlStringProfile URL on Twitter.com
idStringUnique identifier of the user
nameStringDisplay name of the user
isVerifiedBooleanLegacy verification status
isBlueVerifiedBooleanTwitter Blue verification status
verifiedTypeStringType of verification
hasNftAvatarBooleanIndicates if user has an NFT avatar
profilePictureStringURL of the profile picture
coverPictureStringURL of the profile banner/cover
descriptionStringUser's bio text
locationStringUser's location (if provided)
followersNumberCount of followers
followingNumberCount of accounts followed
protectedBooleanIndicates if the account is private
statusStringCurrent status (if set)
canDmBooleanIndicates if user can receive DMs
canMediaTagBooleanIndicates if user can be tagged in media
advertiserAccountTypeStringType of advertiser account
analyticsTypeStringType of analytics enabled
createdAtStringTimestamp of account creation
entitiesObjectURLs and mentions in the bio
fastFollowersCountNumberCount of fast followers
favouritesCountNumberCount of liked tweets
geoEnabledBooleanIndicates if geolocation is enabled
hasCustomTimelinesBooleanIndicates custom timeline usage
hasExtendedProfileBooleanIndicates extended profile features
isTranslatorBooleanIndicates if user is a Twitter translator
mediaCountNumberCount of media posts
profileBackgroundColorStringHex code of profile background color
statusesCountNumberTotal count of tweets/posts
translatorTypeEnumStringType of translator status
withheldInCountriesArrayCountries where account is withheld
affiliatesHighlightedLabelObjectInformation about affiliated labels

Usage Tips

  1. For bulk scraping, use the "Bulk edit" feature to input multiple profile URLs or usernames.
  2. Adjust concurrency settings based on your network capabilities and Twitter's rate limits.
  3. Use proxy configuration for large-scale scraping to avoid IP blocks.

Explore More Scrapers

If you found this Apify Smartbuyglasses Scraper useful, be sure to check out our other powerful scrapers and actors at memo23's Apify profile. We offer a wide range of tools to enhance your web scraping and automation needs across various platforms and use cases.

Support

Additional Services

Developer
Maintained by Community

Actor Metrics

  • 62 monthly users

  • 1 star

  • 97% runs succeeded

  • Created in Oct 2024

  • Modified 14 days ago