Twitter Scraper

Important Notice

Please note that free users are limited to 5 results per run. Consider subscribing to a plan at - https://apify.com/pricing

Supported URLs

• Keyword search e.g., https://twitter.com/search?q=wikipedia&f=live&src=typed_query
• Hashtag search e.g., https://twitter.com/search?q=(%23gpt)&src=typed_query
• Tweet status e.g., https://twitter.com/elonmusk/status/1744812135937396788
• User profile e.g., https://twitter.com/google

Features

• Search and extract public tweets containing data such as favorites count, retweets count, replies count, user mentions, hashtags, urls, symbols, tweet source, geolocation, tweet author, etc.
• Extract a tweet's author info such as username, followers count, friends count, total number of tweets, media count, verified status, etc.
• Search tweets using keywords
• Search tweets using hashtags
• Search tweet trends
• Build your search query and provide the resulting page URL
• No proxy needed

Sample input

{
  "searchTerms": ["cnn"],
  "urls": ["https://twitter.com/search?q=wikipedia&f=live&src=typed_query"],
  "maxTweets": 500,
  "searchMode": "live",
  "maxConcurrency": 1,
  "maxRequestRetries": 3,
  "handle": ["apify"]
}

Sample Response

...
{
	"bookmark_count": 0,
	"bookmarked": false,
	"created_at": "Fri Jul 07 11:11:17 +0000 2023",
	"conversation_id_str": "1677059199765708801",
	"favorite_count": 0,
	"favorited": false,
	"full_text": "@CNN Imagine being so entitled that you deface a countries piece of history. And then send a fake scripted apology. Certainly just as shameful as taking thc oil to Russia?",
	"in_reply_to_screen_name": "CNN",
	"in_reply_to_status_id_str": "1677059199765708801",
	"in_reply_to_user_id_str": "759251",
	"is_quote_status": false,
	"lang": "en",
	"quote_count": 0,
	"reply_count": 0,
	"retweet_count": 0,
	"retweeted": false,
	"user_id_str": "241731823",
	"id_str": "1677274078510567424",
	"user": {
		"can_dm": false,
		"can_media_tag": true,
		"created_at": "Sun Jan 23 00:51:16 +0000 2011",
		"default_profile_image": false,
		"description": "",
		"entities": {
		"user_mentions": [
			{
				"id_str": "17230018",
				"name": "Spotify",
				"screen_name": "Spotify",
				"indices": [
					47,
					55
				]
			}
		],
		"urls": [],
		"hashtags": [],
		"symbols": []
		},
		"fast_followers_count": 0,
		"favourites_count": 37,
		"followers_count": 5,
		"friends_count": 6,
		"has_custom_timelines": false,
		"is_translator": false,
		"listed_count": 0,
		"location": "",
		"media_count": 0,
		"name": "Kenji",
		"normal_followers_count": 5,
		"possibly_sensitive": false,
		"profile_image_url_https": "https://pbs.twimg.com/profile_images/537079467953225728/GaI_XTux_normal.jpeg",
		"screen_name": "Sprken",
		"statuses_count": 174,
		"translator_type": "none",
		"verified": false,
		"want_retweets": false,
		"withheld_in_countries": []
	}
}
...

Limitation & Solution

By default, Twitter returns ~800 results per search query/URL. To resolve this limitation, you can search for the same keyword multiple times but using different date ranges. You can build the search query and make e.g., 3 API calls in the format:

• https://twitter.com/search?q=wikipedia%20until%3A2022-01-31%20since%3A2022-01-01&src=typed_query&f=live - Searches for the keyword wikipedia from 2022-01-01 until 2022-01-31
• https://twitter.com/search?q=wikipedia%20until%3A2022-02-28%20since%3A2022-02-01&src=typed_query&f=live - Searches for the keyword wikipedia from 2022-02-01 until 2022-02-28
• https://twitter.com/search?q=wikipedia%20until%3A2022-03-31%20since%3A2022-03-01&src=typed_query&f=live - Searches for the keyword wikipedia from 2022-03-01 until 2022-03-31