Actor picture

Twitter Scraper

vdrmota/twitter-scraper

Scrape any Twitter user profile. Extract tweets, retweets, replies, favorites, and conversation threads with no Twitter API limits. Download your data as HTML table, JSON, CSV, Excel, XML, and RSS feed.

Author's avatarVojta Drmota
  • Modified
  • Used by781 users
  • Used266,629 times
Actor picture
Twitter Scraper

Search terms

searchTerms

Optional

array

Search for an specific terms and extract tweets for that term.

Search mode

searchMode

Optional

string

Search mode changes the way the data is received

Options:

"live", "user", "image", "video"

Start URLs

startUrls

Optional

array

If you have a file or a url list of twitter urls, you provide them here

Tweets newer than

toDate

Optional

string

Will get tweets that are newer than this date. Can be used in conjunction with 'Tweets older than' to create specific date slices. Can use specific dates, such as YYYY-MM-DD or relative ones, like '1 month' or '2 days'

Tweets older than

fromDate

Optional

string

Will start getting tweets from this date and older. Can be used in conjunction 'Tweets newer than'. Can use specific dates, such as YYYY-MM-DD or relative ones, like '1 month' or '2 days'

Maximum number of tweets

tweetsDesired

Optional

integer

Maximum number of tweets to retrieve. Twitter have a default limit of around 3200 tweets. Check the README for workarounds.

Add user information

addUserInfo

Optional

boolean

Appends an object to each tweet containing the user information. You can decrease the size of your dataset by turning this off.

Tweet types

mode

Optional

string

Select the tweet types to get. Only tweets or tweets and replies for the selected handle when visiting profiles.

Options:

"own", "replies"

List of usernames to scrape

handle

Optional

array

The twitter handles of the profiles you want to scrape. Shortcut instead of inputting https://twitter.com/username urls

Proxy configuration

proxyConfig

Required

object

It's required to use proxies when running on the platform.

Extend Output Function

extendOutputFunction

Optional

string

Add or remove properties on the output object or omit the output returning null

Extend Scraper Function

extendScraperFunction

Optional

string

Advanced function that allows you to extend the default scraper functionality, allowing you to manually perform actions on the page

Custom data

customData

Optional

object

Any data that you want to have available inside the Extend Output/Scraper Function

Max timeout seconds

handlePageTimeoutSecs

Optional

integer

Max timeout for the handlePageFunction. Can be increased for long running processes

Stealth

stealth

Optional

boolean

Enabling stealth allows to decrease the chance of your scrape being detected as an automated process. Recommended to enable if you're providing your login cookies.

Login cookies

initialCookies

Optional

array

Your login cookies will be used to bypass the login wall. See README for instructions.