Twitter Scraper

Important Notice

Please note that free users are limited to 5 results per run. Consider subscribing to a plan at - https://apify.com/pricing

Supported URLs

• Keyword search e.g., https://twitter.com/search?q=wikipedia&f=live&src=typed_query
• Hashtag search e.g., https://twitter.com/search?q=(%23gpt)&src=typed_query
• Tweet status e.g., https://twitter.com/elonmusk/status/1744812135937396788
• User profile e.g., https://twitter.com/google

Features

• Search and extract public tweets containing data such as favorites count, retweets count, replies count, user mentions, hashtags, urls, symbols, tweet source, geolocation, tweet author, etc.
• Extract a tweet's author info such as username, followers count, friends count, total number of tweets, media count, verified status, etc.
• Search tweets using keywords
• Search tweets using hashtags
• Search tweet trends
• Build your search query and provide the resulting page URL
• No proxy needed

Sample input

1{
2  "searchTerms": ["cnn"],
3  "urls": ["https://twitter.com/search?q=wikipedia&f=live&src=typed_query"],
4  "maxTweets": 500,
5  "searchMode": "live",
6  "maxConcurrency": 1,
7  "maxRequestRetries": 3,
8  "handle": ["apify"]
9}

Sample Response

1...
2{
3	"bookmark_count": 0,
4	"bookmarked": false,
5	"created_at": "Fri Jul 07 11:11:17 +0000 2023",
6	"conversation_id_str": "1677059199765708801",
7	"favorite_count": 0,
8	"favorited": false,
9	"full_text": "@CNN Imagine being so entitled that you deface a countries piece of history. And then send a fake scripted apology. Certainly just as shameful as taking thc oil to Russia?",
10	"in_reply_to_screen_name": "CNN",
11	"in_reply_to_status_id_str": "1677059199765708801",
12	"in_reply_to_user_id_str": "759251",
13	"is_quote_status": false,
14	"lang": "en",
15	"quote_count": 0,
16	"reply_count": 0,
17	"retweet_count": 0,
18	"retweeted": false,
19	"user_id_str": "241731823",
20	"id_str": "1677274078510567424",
21	"user": {
22		"can_dm": false,
23		"can_media_tag": true,
24		"created_at": "Sun Jan 23 00:51:16 +0000 2011",
25		"default_profile_image": false,
26		"description": "",
27		"entities": {
28		"user_mentions": [
29			{
30				"id_str": "17230018",
31				"name": "Spotify",
32				"screen_name": "Spotify",
33				"indices": [
34					47,
35					55
36				]
37			}
38		],
39		"urls": [],
40		"hashtags": [],
41		"symbols": []
42		},
43		"fast_followers_count": 0,
44		"favourites_count": 37,
45		"followers_count": 5,
46		"friends_count": 6,
47		"has_custom_timelines": false,
48		"is_translator": false,
49		"listed_count": 0,
50		"location": "",
51		"media_count": 0,
52		"name": "Kenji",
53		"normal_followers_count": 5,
54		"possibly_sensitive": false,
55		"profile_image_url_https": "https://pbs.twimg.com/profile_images/537079467953225728/GaI_XTux_normal.jpeg",
56		"screen_name": "Sprken",
57		"statuses_count": 174,
58		"translator_type": "none",
59		"verified": false,
60		"want_retweets": false,
61		"withheld_in_countries": []
62	}
63}
64...

Limitation & Solution

By default, Twitter returns ~800 results per search query/URL. To resolve this limitation, you can search for the same keyword multiple times but using different date ranges. You can build the search query and make e.g., 3 API calls in the format:

• https://twitter.com/search?q=wikipedia%20until%3A2022-01-31%20since%3A2022-01-01&src=typed_query&f=live - Searches for the keyword wikipedia from 2022-01-01 until 2022-01-31
• https://twitter.com/search?q=wikipedia%20until%3A2022-02-28%20since%3A2022-02-01&src=typed_query&f=live - Searches for the keyword wikipedia from 2022-02-01 until 2022-02-28
• https://twitter.com/search?q=wikipedia%20until%3A2022-03-31%20since%3A2022-03-01&src=typed_query&f=live - Searches for the keyword wikipedia from 2022-03-01 until 2022-03-31