Actor picture

Twitter Scraper

vdrmota/twitter-scraper

Scrape any Twitter user profile. Creates an unofficial Twitter API to extract tweets, retweets, replies, favorites, and conversation threads with no Twitter API limits. Download your data as HTML table, JSON, CSV, Excel, XML, and use it in spreadsheets, applications, reports, and databases.

No credit card required

Author's avatarVojta Drmota
  • Modified
  • Users2,494
  • Runs429,867

What search terms do you want to scrape?

searchTerms

Optional

array

If you add search terms, the scraper will find and extract tweets that mention those terms. Alternatively, see further down to scrape by Twitter handle or URL.

Do you want to filter by content?

searchMode

Optional

string

This setting will change how the data is received by the scraper.

Options:

"live", "user", "image", "video"

Do you want to scrape replies in addition to tweets?

mode

Optional

string

You can choose to only scrape a user's own tweets or you can choose to also scrape the user's own tweets and the user's replies to other users.

Options:

"own", "replies"

Set the maximum number of tweets

tweetsDesired

Optional

integer

This value lets you set the maximum number of tweets to retrieve. Twitter has a default limit of around 3,200 tweets. Check the README for workarounds.

Add user information

addUserInfo

Optional

boolean

Appends an object to each tweet containing the user information. You can decrease the size of your dataset by turning this off.

Do you want to scrape by Twitter handle?

handle

Optional

array

You can add the twitter handles of specific profiles you want to scrape. This is a shortcut so that you don't have to add full username URLs like https://twitter.com/username.

Do you want to scrape by Twitter URL?

startUrls

Optional

array

This lets you tell the scraper where to start. You can enter Twitter URLs one by one. You can also link to or upload a text file with a list of URLs.

Tweets newer than

toDate

Optional

string

Scrape tweets newer than this date. You can use this in conjunction with 'Tweets older than' to create a limited time slice.

Tweets older than

fromDate

Optional

string

Scrape tweets from this date and before. You can use this in conjunction with 'Tweets newer than'to create a limited time slice.

Proxy configuration

proxyConfig

Required

object

This is required if you want to use Apify Proxy.

Extend Output Function

extendOutputFunction

Optional

string

Add or remove properties on the output object or omit the output returning null

Extend Scraper Function

extendScraperFunction

Optional

string

Advanced function that allows you to extend the default scraper functionality, allowing you to manually perform actions on the page

Custom data

customData

Optional

object

Any data that you want to have available inside the Extend Output/Scraper Function

Max timeout seconds

handlePageTimeoutSecs

Optional

integer

Max timeout for the handlePageFunction. Can be increased for long running processes

Max request retries

maxRequestRetries

Optional

integer

Set the max request retries

Scrolling idle seconds

maxIdleTimeoutSecs

Optional

integer

Configures how many seconds of no data received will be considered done

Debug log

debugLog

Optional

boolean

Enable debug log

Login cookies

initialCookies

Optional

array

Your login cookies will be used to bypass the login wall. Check the README for detailed instructions.