Actor - Twitter Search Scraper

Since Twitter doesn't provide a good and free API, this actor should help you to retrieve data from it.

Features

• Scrape user detail - Scrape any user-related information from any search result such as friends, followers, following, verified account, location, profile image, profile banner, profile URL, date of creation, and so on.

• Scrape tweet - You can retrieve all the tweets from any keyword with all the details. Language, sensitive language, reply, quote, retweet, pinned, retweeted, and all sorts of Tweet-related information.

• Scrape statistics - Gather all the statistical information of a tweet.

Why use this actor?

This actor is extremely fast and optimized. It'll scrape tweets around 31 times faster than the other equivalent scrapers. Therefore you will consume fewer resources and it will be cheaper to use it.

Bugs, fixes, updates, and changelog

This scraper is under active development. If you have any feature requests you can create an issue from here.

Input Parameters

The input of this scraper should be JSON containing the list of pages on the Twitter Search Scraper that should be visited. Possible fields are:

• keywords: (Required) (Array) List of keywords you want to scrape from Twitter.

• maxItems: (Optional) (Number) You can limit scraped items. This should be useful when you search through the big lists or search results.

• proxy: (Required) (Proxy Object) Proxy configuration.

This solution requires the use of Proxy servers, either your own proxy servers or you can use Apify Proxy.

Tip

When you want to have a scrape over a specific keyword or hashtag, just copy and paste the keyword as one of the keyword.

Compute Unit Consumption

The actor is optimized to run blazing fast and scrape many tweets as possible. Therefore, it forefronts all tweet detail requests. If the actor doesn't block very often it'll scrape 100 tweets in 20 seconds with ~0.02-0.025 compute units.

Twitter Search Scraper Input example

1{
2  "proxy":{
3    "useApifyProxy":true
4  },
5  "maxItems": 10,
6  "keywords":[
7    "#binance",
8    "ukraine"
9  ]
10}

During the Run

During the run, the actor will output messages letting you know what is going on. Each message always contains a short label specifying which page from the provided list is currently specified. When items are loaded from the page, you should see a message about this event with a loaded item count and total item count for each page.

If you provide incorrect input to the actor, it will immediately stop with a failure state and output an explanation of what is wrong.

Twitter Search Scraper Export

During the run, the actor stores results into a dataset. Each item is a separate item in the dataset.

You can manage the results in any language (Python, PHP, Node JS/NPM). See the FAQ or our API reference to learn more about getting results from this Twitter Search actor.

Scraped Tweets Properties

The structure of each tweet in Twitter Search Scraper looks like this:

Tweet Detail

1{
2    "created_at": "Sat Nov 19 12:05:12 +0000 2022",
3    "id": 1593938439887618000,
4    "id_str": "1593938439887618054",
5    "full_text": "Daha önce #MeralAkşener iyine çağırdı yok dedi, #AhmetKaya 'nın mezarını Türkiye'ye getirebiliriz dedi eşi Gülten Kaya yok dedi.Her seçim zamanı aynı repertuarı kullanmasan mı ? #izmirdeprem",
6    "truncated": false,
7    "display_text_range": [
8        0,
9        190
10    ],
11    "entities": {
12        "hashtags": [
13            {
14                "text": "MeralAkşener",
15                "indices": [
16                    10,
17                    23
18                ]
19            },
20            {
21                "text": "AhmetKaya",
22                "indices": [
23                    48,
24                    58
25                ]
26            },
27            {
28                "text": "izmirdeprem",
29                "indices": [
30                    178,
31                    190
32                ]
33            }
34        ],
35        "symbols": [],
36        "user_mentions": [],
37        "urls": []
38    },
39    "source": "<a href=\"http://twitter.com/download/android\" rel=\"nofollow\">Twitter for Android</a>",
40    "in_reply_to_status_id": null,
41    "in_reply_to_status_id_str": null,
42    "in_reply_to_user_id": null,
43    "in_reply_to_user_id_str": null,
44    "in_reply_to_screen_name": null,
45    "user_id": 1202948684297724000,
46    "user_id_str": "1202948684297723905",
47    "geo": null,
48    "coordinates": null,
49    "place": null,
50    "contributors": null,
51    "is_quote_status": false,
52    "retweet_count": 0,
53    "favorite_count": 0,
54    "reply_count": 0,
55    "quote_count": 0,
56    "conversation_id": 1593938439887618000,
57    "conversation_id_str": "1593938439887618054",
58    "favorited": false,
59    "retweeted": false,
60    "lang": "tr",
61    "supplemental_language": null,
62    "ext_edit_control": {
63        "initial": {
64            "edit_tweet_ids": [
65                "1593938439887618054"
66            ],
67            "editable_until_msecs": "1668861312000",
68            "edits_remaining": "5",
69            "is_edit_eligible": true
70        }
71    },
72    "ext": {
73        "unmentionInfo": {
74            "r": {
75                "ok": {}
76            },
77            "ttl": -1
78        },
79        "superFollowMetadata": {
80            "r": {
81                "ok": {}
82            },
83            "ttl": -1
84        },
85        "editControl": {
86            "r": {
87                "ok": {
88                    "initial": {
89                        "editTweetIds": [
90                            "1593938439887618054"
91                        ],
92                        "editableUntilMsecs": "1668861312000",
93                        "editsRemaining": "5",
94                        "isEditEligible": true
95                    }
96                }
97            },
98            "ttl": -1
99        }
100    },
101    "user": {
102        "id": 1202948684297724000,
103        "id_str": "1202948684297723905",
104        "name": "Mete Han",
105        "screen_name": "Metehanatam",
106        "location": "",
107        "description": "Atatürk - Fenerbahçe -Hayvansever - Rakı Sever",
108        "url": null,
109        "entities": {
110            "description": {
111                "urls": []
112            }
113        },
114        "protected": false,
115        "followers_count": 40,
116        "fast_followers_count": 0,
117        "normal_followers_count": 40,
118        "friends_count": 224,
119        "listed_count": 0,
120        "created_at": "Fri Dec 06 13:51:37 +0000 2019",
121        "favourites_count": 241,
122        "utc_offset": null,
123        "time_zone": null,
124        "geo_enabled": true,
125        "verified": false,
126        "statuses_count": 799,
127        "media_count": 251,
128        "lang": null,
129        "contributors_enabled": false,
130        "is_translator": false,
131        "is_translation_enabled": false,
132        "profile_background_color": "F5F8FA",
133        "profile_background_image_url": null,
134        "profile_background_image_url_https": null,
135        "profile_background_tile": false,
136        "profile_image_url": "http://pbs.twimg.com/profile_images/1503755569475297285/uZLS4w8m_normal.jpg",
137        "profile_image_url_https": "https://pbs.twimg.com/profile_images/1503755569475297285/uZLS4w8m_normal.jpg",
138        "profile_banner_url": "https://pbs.twimg.com/profile_banners/1202948684297723905/1651961644",
139        "profile_image_extensions_sensitive_media_warning": null,
140        "profile_image_extensions_media_availability": null,
141        "profile_image_extensions_alt_text": null,
142        "profile_image_extensions_media_color": {
143            "palette": [
144                {
145                    "rgb": {
146                        "red": 45,
147                        "green": 29,
148                        "blue": 26
149                    },
150                    "percentage": 37.92
151                },
152                {
153                    "rgb": {
154                        "red": 196,
155                        "green": 167,
156                        "blue": 146
157                    },
158                    "percentage": 25.34
159                },
160                {
161                    "rgb": {
162                        "red": 191,
163                        "green": 131,
164                        "blue": 83
165                    },
166                    "percentage": 6.52
167                },
168                {
169                    "rgb": {
170                        "red": 110,
171                        "green": 44,
172                        "blue": 43
173                    },
174                    "percentage": 4.74
175                },
176                {
177                    "rgb": {
178                        "red": 192,
179                        "green": 52,
180                        "blue": 60
181                    },
182                    "percentage": 4.72
183                }
184            ]
185        },
186        "profile_image_extensions": {
187            "mediaStats": {
188                "r": {
189                    "missing": null
190                },
191                "ttl": -1
192            }
193        },
194        "profile_banner_extensions_sensitive_media_warning": null,
195        "profile_banner_extensions_media_availability": null,
196        "profile_banner_extensions_alt_text": null,
197        "profile_banner_extensions_media_color": {
198            "palette": [
199                {
200                    "rgb": {
201                        "red": 41,
202                        "green": 50,
203                        "blue": 63
204                    },
205                    "percentage": 69.82
206                },
207                {
208                    "rgb": {
209                        "red": 98,
210                        "green": 110,
211                        "blue": 133
212                    },
213                    "percentage": 13.17
214                },
215                {
216                    "rgb": {
217                        "red": 107,
218                        "green": 83,
219                        "blue": 75
220                    },
221                    "percentage": 8.99
222                },
223                {
224                    "rgb": {
225                        "red": 143,
226                        "green": 149,
227                        "blue": 140
228                    },
229                    "percentage": 2.25
230                },
231                {
232                    "rgb": {
233                        "red": 176,
234                        "green": 115,
235                        "blue": 67
236                    },
237                    "percentage": 1.49
238                }
239            ]
240        },
241        "profile_banner_extensions": {
242            "mediaStats": {
243                "r": {
244                    "missing": null
245                },
246                "ttl": -1
247            }
248        },
249        "profile_link_color": "1DA1F2",
250        "profile_sidebar_border_color": "C0DEED",
251        "profile_sidebar_fill_color": "DDEEF6",
252        "profile_text_color": "333333",
253        "profile_use_background_image": true,
254        "has_extended_profile": false,
255        "default_profile": true,
256        "default_profile_image": false,
257        "pinned_tweet_ids": [
258            1506948451191472000
259        ],
260        "pinned_tweet_ids_str": [
261            "1506948451191472128"
262        ],
263        "has_custom_timelines": false,
264        "can_dm": null,
265        "following": null,
266        "follow_request_sent": null,
267        "notifications": null,
268        "muting": null,
269        "blocking": null,
270        "blocked_by": null,
271        "want_retweets": null,
272        "advertiser_account_type": "none",
273        "advertiser_account_service_levels": [],
274        "profile_interstitial_type": "",
275        "business_profile_state": "none",
276        "translator_type": "none",
277        "withheld_in_countries": [],
278        "followed_by": null,
279        "ext_is_blue_verified": false,
280        "ext_has_nft_avatar": false,
281        "ext": {
282            "highlightedLabel": {
283                "r": {
284                    "ok": {}
285                },
286                "ttl": -1
287            },
288            "hasNftAvatar": {
289                "r": {
290                    "ok": false
291                },
292                "ttl": -1
293            },
294            "superFollowMetadata": {
295                "r": {
296                    "ok": {
297                        "superFollowEligible": false,
298                        "superFollowing": false,
299                        "superFollowedBy": false,
300                        "exclusiveTweetFollowing": false,
301                        "privateSuperFollowing": false
302                    }
303                },
304                "ttl": -1
305            }
306        },
307        "require_some_consent": false
308    }
309}

Contact

Please visit us through epctex.com to see all the products that are available for you. If you are looking for any custom integration or so, please reach out to us through the chat box in epctex.com. In need of support? devops@epctex.com is at your service.