Actor picture

Twitter Scraper

vdrmota/twitter-scraper

Scrape any Twitter user profile. Creates an unofficial Twitter API to extract tweets, retweets, replies, favorites, and conversation threads with no Twitter API limits. Download your data as HTML table, JSON, CSV, Excel, XML, and use it in spreadsheets, applications, reports, and databases.

No credit card required

Author's avatarVojta Drmota
  • Modified
  • Users1,387
  • Runs366,087

Search terms

searchTerms

Optional

array

Search for an specific terms and extract tweets for that term.

Usernames

handle

Optional

array

The twitter handles of the profiles you want to scrape. Shortcut instead of inputting https://twitter.com/username urls

Start URLs

startUrls

Optional

array

If you have a file or a url list of twitter urls, you provide them here

Search mode

searchMode

Optional

string

Search mode changes the way the data is received

Options:

"live", "user", "image", "video"

Tweet types

mode

Optional

string

Select the tweet types to get. Only tweets or tweets and replies for the selected handle when visiting profiles.

Options:

"own", "replies"

Maximum number of tweets

tweetsDesired

Optional

integer

Maximum number of tweets to retrieve. Twitter have a default limit of around 3200 tweets. Check the README for workarounds.

Add user information

addUserInfo

Optional

boolean

Appends an object to each tweet containing the user information. You can decrease the size of your dataset by turning this off.

Tweets newer than

toDate

Optional

string

Will get tweets that are newer than this date. Can be used in conjunction with 'Tweets older than' to create specific date slices. Can use specific dates, such as YYYY-MM-DD or relative ones, like '1 month' or '2 days'

Tweets older than

fromDate

Optional

string

Will start getting tweets from this date and older. Can be used in conjunction 'Tweets newer than'. Can use specific dates, such as YYYY-MM-DD or relative ones, like '1 month' or '2 days'

Proxy configuration

proxyConfig

Required

object

It's required to use proxies when running on the platform.

Extend Output Function

extendOutputFunction

Optional

string

Add or remove properties on the output object or omit the output returning null

Extend Scraper Function

extendScraperFunction

Optional

string

Advanced function that allows you to extend the default scraper functionality, allowing you to manually perform actions on the page

Custom data

customData

Optional

object

Any data that you want to have available inside the Extend Output/Scraper Function

Max timeout seconds

handlePageTimeoutSecs

Optional

integer

Max timeout for the handlePageFunction. Can be increased for long running processes

Stealth

stealth

Optional

boolean

Enabling stealth allows to decrease the chance of your scrape being detected as an automated process. Recommended to enable if you're providing your login cookies.

Max request retries

maxRequestRetries

Optional

integer

Set the max request retries

Scrolling idle seconds

maxIdleTimeoutSecs

Optional

integer

Configures how many seconds of no data received to consider it done

Debug log

debugLog

Optional

boolean

Enable debug log

Login cookies

initialCookies

Optional

array

Your login cookies will be used to bypass the login wall. See README for instructions.