Twitter Profiles and Tweets Scraper

Overview

This powerful scraper allows you to extract comprehensive data from Twitter (X) profiles and their tweets. It's designed to work with various input formats, including full profile URLs, usernames, and even Instagram profile URLs.

Features

• Scrape detailed profile information
• Extract tweets with comprehensive metadata
• Supports multiple input URLs, including Twitter handles and profile URLs.
• Username-only input support
• Configurable concurrency and retry mechanisms
• Optional inclusion of posts/tweets
• Proxy configuration for enhanced reliability
• Extracts tweets, user information, engagement metrics, tweet sources, and more.

Input Configuration

1. Start URLs:

   • Full profile URLs: https://x.com/elonmusk or https://twitter.com/elonmusk
   • Usernames: adrian_horning_
   • Instagram URLs (for cross-platform profiles): https://www.instagram.com/robertsyfert/

2. Max Concurrency: Set maximum concurrent requests (default: 10)

3. Min Concurrency: Set minimum concurrent requests (default: 1)

4. Max Request Retries: Set maximum retry attempts for failed requests (default: 100)

Input Data

Here's an example of how to set up a Twitter profile scraping task:

1{
2    "startUrls": [
3        "https://x.com/elonmusk",
4        "elonmusk",
5        "https://www.instagram.com/robertsyfert/"
6    ],
7    "maxConcurrency": 10,
8    "minConcurrency": 1,
9    "maxRequestRetries": 100
10}

Output Structure

1{
2  "type": "tweet",
3  "id": "1519480761749016577",
4  "url": "https://x.com/elonmusk/status/1519480761749016577",
5  "twitterUrl": "https://twitter.com/elonmusk/status/1519480761749016577",
6  "text": "Next I'm buying Coca-Cola to put the cocaine back in",
7  "retweetCount": 620127,
8  "replyCount": 179263,
9  "likeCount": 4434571,
10  "quoteCount": 166686,
11  "createdAt": "Thu Apr 28 00:56:58 +0000 2022",
12  "lang": "en",
13  "quoteId": null,
14  "bookmarkCount": 21208,
15  "isReply": false,
16  "card": {},
17  "place": {},
18  "source": "<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>",
19  "author": {
20    "type": "user",
21    "userName": "elonmusk",
22    "url": "https://x.com/elonmusk",
23    "twitterUrl": "https://twitter.com/elonmusk",
24    "id": "44196397",
25    "name": "Elon Musk",
26    "isVerified": true,
27    "isBlueVerified": true,
28    "verifiedType": "none",
29    "hasNftAvatar": false,
30    "profilePicture": "https://pbs.twimg.com/profile_images/1845482317860450309/OrD0ovmf_normal.jpg",
31    "coverPicture": "https://pbs.twimg.com/profile_banners/44196397/1726163678",
32    "description": "Read @America to understand why I'm supporting Trump for President",
33    "location": "",
34    "followers": 201720551,
35    "following": 787,
36    "protected": false,
37    "status": "",
38    "canDm": false,
39    "canMediaTag": false,
40    "advertiserAccountType": "promotable_user",
41    "analyticsType": "enabled",
42    "createdAt": "Tue Jun 02 20:12:29 +0000 2009",
43    "entities": {
44      "description": {
45        "urls": []
46      },
47      "url": {
48        "urls": [
49          {
50            "display_url": "TheAmericaPAC.org",
51            "expanded_url": "http://TheAmericaPAC.org",
52            "url": "https://t.co/DjyKIO6ePx",
53            "indices": [
54              0,
55              23
56            ]
57          }
58        ]
59      }
60    },
61    "fastFollowersCount": 0,
62    "favouritesCount": 81232,
63    "geoEnabled": false,
64    "hasCustomTimelines": true,
65    "hasExtendedProfile": false,
66    "isTranslator": false,
67    "mediaCount": 2592,
68    "profileBackgroundColor": "C0DEED",
69    "statusesCount": 54347,
70    "translatorTypeEnum": "None",
71    "withheldInCountries": [],
72    "affiliatesHighlightedLabel": {
73      "label": {
74        "url": {
75          "url": "https://twitter.com/X",
76          "urlType": "DeepLink"
77        },
78        "badge": {
79          "url": "https://pbs.twimg.com/profile_images/1683899100922511378/5lY42eHs_bigger.jpg"
80        },
81        "description": "X",
82        "userLabelType": "BusinessLabel",
83        "userLabelDisplayType": "Badge"
84      }
85    }
86  },
87  "extendedEntities": {},
88  "isRetweet": false,
89  "isQuote": false,
90  "media": []
91}

Output Format

The scraper provides rich, structured data for both profiles and tweets. Here's a detailed breakdown of the output fields:

Tweet Object

Field	Type	Description
type	String	Always "tweet" for tweet objects
id	String	Unique identifier of the tweet
url	String	Full URL to the tweet on X.com
twitterUrl	String	Full URL to the tweet on Twitter.com
text	String	Content of the tweet
retweetCount	Number	Count of retweets
replyCount	Number	Count of replies
likeCount	Number	Count of likes
quoteCount	Number	Count of quote tweets
createdAt	String	Timestamp of tweet creation
lang	String	Language code of the tweet
quoteId	String	null
bookmarkCount	Number	Count of bookmarks
isReply	Boolean	Indicates if the tweet is a reply
card	Object	Additional card information (e.g., for links)
place	Object	Location information, if available
source	String	Application used to post the tweet
isRetweet	Boolean	Indicates if the tweet is a retweet
isQuote	Boolean	Indicates if the tweet is a quote tweet
media	Array	Contains media objects (images, videos)
extendedEntities	Object	Additional entities (e.g., extended media info)

Author Object (User Profile)

Field	Type	Description
type	String	Always "user" for profile objects
userName	String	Twitter handle of the user
url	String	Profile URL on X.com
twitterUrl	String	Profile URL on Twitter.com
id	String	Unique identifier of the user
name	String	Display name of the user
isVerified	Boolean	Legacy verification status
isBlueVerified	Boolean	Twitter Blue verification status
verifiedType	String	Type of verification
hasNftAvatar	Boolean	Indicates if user has an NFT avatar
profilePicture	String	URL of the profile picture
coverPicture	String	URL of the profile banner/cover
description	String	User's bio text
location	String	User's location (if provided)
followers	Number	Count of followers
following	Number	Count of accounts followed
protected	Boolean	Indicates if the account is private
status	String	Current status (if set)
canDm	Boolean	Indicates if user can receive DMs
canMediaTag	Boolean	Indicates if user can be tagged in media
advertiserAccountType	String	Type of advertiser account
analyticsType	String	Type of analytics enabled
createdAt	String	Timestamp of account creation
entities	Object	URLs and mentions in the bio
fastFollowersCount	Number	Count of fast followers
favouritesCount	Number	Count of liked tweets
geoEnabled	Boolean	Indicates if geolocation is enabled
hasCustomTimelines	Boolean	Indicates custom timeline usage
hasExtendedProfile	Boolean	Indicates extended profile features
isTranslator	Boolean	Indicates if user is a Twitter translator
mediaCount	Number	Count of media posts
profileBackgroundColor	String	Hex code of profile background color
statusesCount	Number	Total count of tweets/posts
translatorTypeEnum	String	Type of translator status
withheldInCountries	Array	Countries where account is withheld
affiliatesHighlightedLabel	Object	Information about affiliated labels

Usage Tips

1. For bulk scraping, use the "Bulk edit" feature to input multiple profile URLs or usernames.
2. Adjust concurrency settings based on your network capabilities and Twitter's rate limits.
3. Use proxy configuration for large-scale scraping to avoid IP blocks.

Explore More Scrapers

If you found this Apify Smartbuyglasses Scraper useful, be sure to check out our other powerful scrapers and actors at memo23's Apify profile. We offer a wide range of tools to enhance your web scraping and automation needs across various platforms and use cases.

Support

• For issues or feature requests, please use the Issues section of this actor.
• If you need customization or have questions, feel free to contact the author:
  • Author's website: https://muhamed-didovic.github.io/
  • Email: muhamed.didovic@gmail.com

Additional Services

• Request customization or whole dataset: muhamed.didovic@gmail.com
• If you need anything else scraped, or this actor customized, email: muhamed.didovic@gmail.com
• For API services of this scraper (no Apify fee, just usage fee for the API), contact: muhamed.didovic@gmail.com
• Email: muhamed.didovic@gmail.com