Pay $3.50 for 1,000 posts

Twitter Scraper

quacker/twitter-scraper

Pay $3.50 for 1,000 posts

Scrape tweets from any Twitter user profile. Top Twitter API alternative to scrape Twitter hashtags, threads, replies, followers, images, videos, statistics, and Twitter history. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

2024-03-07

BREAKING CHANGES

For profiles and tweets that are not found, you will now get an item in the dataset with the URL property and the reason this profile/tweet has not been scraped

2024-01-04

Fix:

Fixed an issue handling tweets URLs

2023-10-17

Features:

Support x.com urls

2023-08-10

Changes:

Removed toDate input option because now tweets are not ordered and there is limit of 100 tweets per profile (By Twitter)

2023-08-03

Changes:

Removed fromDate and relativeFromDate input options

2023-07-12

Fixes:

Fixed issue where not all search urls were working

2023-07-10

Changes:

Now you can scrape individual tweets by providing the tweet url and/or user data by providing the tweet url

2023-06-30

BREAKING CHANGES: On 30 June 2023, Twitter put all content behind login, except for single tweets, so scraping publicly available Twitter content is no longer possible.

You can have a look at https://apify.com/shanes/tweet-flash or https://apify.com/web.harvester/easy-twitter-search-scraperactor that don't require login.

2023-06-22

Changes:

Deprecated fromDate and relativeFromDate input options, they will be removed in a few weeks

2023-06-19

Changes:

Removed skipPromotedTweets option
Update repliesDepth option to be consistent for profiles and tweets

Fixes:

Fixed issue where the actor would get stuck on some tweets that are not available
Fixed issue with scraping replies of replies for profiles

2023-06-05

Changes:

Disabled skipPromotedTweets due to an issue
Removed maxRequestRetries, handlePageTimeoutSecs, maxConcurrency, and maxIdleTimeoutSecs fields from input schema
The actor now uses the cheerioCrawler by default, which is more reliable and efficient than the puppeteerCrawler
Changed the option collectOriginalTweetOnly to collectOriginalTweetOnly, since it's not only for retweets, it also applies to quoted tweets
If the tweetsDesired is set to 0, the actor will only scrape user info and won't scrape any tweets even if addUserInfo is set to false

2023-05-30

Changes:

BREAKING CHANGES: Completely removed extendScraperFunction function from the actor.
Deprecated extendOutputFunction function, it will be supported for a few weeks before getting removed completely.

2023-05-29

Changes:

Add an experimental option useNewProfileScraper that allows scraping multiple profiles at once faster and more efficiently
Add an experimental option useNewTweetsScraper that allows scraping multiple Tweets at once faster and more efficiently
Add an option skipPromotedTweets that allows the users to skip promoted tweets

Features:

Add is_thread, is_root_thread, and root_thread_url to the output
Add includeThreadsOnly option to include only threads in the output
Add username and user_id fields to output, if addUserInfo is false

2023-05-09

Features:

Add repliesDepth option to scrape replies of replies bases on the depth provided

Changes:

Removed addTweetViewCount option, now the view count is always scraped

2023-05-05

Features:

For retweets now the retweet object will contain the original tweet, and the tweet object will contain the retweet
Add skipRetweets option to skip retweets
Add collectOriginalTweetOnly option to collect only the original retweet

2023-04-28

Features:

Now truncated tweets full text is scraped
Add is_truncated to the output
Add the option tweetsLanguage to search tweets by language
Add the option keywordsSearchType to search for tweets that contain all the keywords or any of them or the exact phrase or none of them
Add relativeToDate and relativeFromDate to the input to search for tweets using relative dates instead of absolute dates

Fixes:

Fixed an issue were quoted tweets were not always scraped
Fixed an issue with requestsFromUrl
Fixed issue where view_count would be undefined for retweets

2023-04-25

Features:

Add quote_count to the output

2023-04-09

Features:

Add video_url or gif_url to media object in the output

2023-04-07

Fixes:

Fixed issue where the browser would get stuck when there a couple of tweets only

Changes:

Removed browserFallback option and updated the method of scraping headers that is faster, doesn't require a lot of resources and doesn't depend on another actor

2023-04-05

Fixes:

Fixed issue with extracting info from quote tweets that are retweets

2023-04-04

Fixes:

Some quoted tweets were not scraped

2023-04-03

Features:

Add is_retweet to the output

Fixes:

Fixed issue where pinned tweets are not scraped
Filter replies in advanced search

2023-03-29

Features:

Add new option browserFallback, this option enables a fallback to browser-based scraping if Cheerio requests fail (This will use another actor in order to function = more resources). When enabled, the actor will attempt to use the browser to retrieve tweets and provide the results to Cheerio for parsing. This process will occur automatically for every new request, improving the actor's ability to scrape tweets.
Now you will be able to scrape tweets indefinitely, the actor will try scrape all the tweets available for the given url, and it will stop when it reaches the tweetsDesired limit, that is if you enabled the new option browserFallback.

Changes:

Remove tweetsDesired limit, now the you can scrape as many tweets as you want
Add runId to the key-value store to prevent conflicts with other runs

Fixes:

Fixed reties to scrape tweets that are deleted
Handle issue where the twitter website would get stuck
Fixed issue where view_count would be undefined
Fixed issue where the actor would get stuck on some tweets that are not available

2023-03-15

Features:

Added useAdvancedSearch option to use the advanced search instead of the regular search for content type Search, it works with fromDate, toDate, searchTerms and handles (usernames). It's disabled by default, and it doesn't scrape retweets.

2023-03-02

Features:

Added replying_to_tweet to the output, which is the link to the tweet that the current tweet is replying to
Added is_quote_tweet and quoted_tweet to the output, which is the tweet that the current tweet is quoting

Changes:

Small code refactoring.

Fixes:

Fixed issue where twitter sometimes return not tweets for a page, which was causing the request to finish without collecting all the tweets.

2023-02-02

Changes:

The Actor now uses Cheerio Crawler for content type 'People'
Improved cheerioCrawler stability.

Fixes:

Fixed issue where the actor get stuck on private profiles
Fixed issue where some requests where not handled properly or not added to the queue

2023-01-25

Changes:

Increased default maxRequestRetries to 6

2023-01-23

Features:

Add profilesDesired option to limit the number of scraped profiles

Changes:

Improved cheerio scraper by allowing it to scrape tweets using any content type
Improved logging for info and errors
The actor now uses the cheerioCrawler by default, which is more reliable than the puppeteerCrawler
Removed the tweetsDesired max limit, which was 3200 tweets

2023-01-21

Changes:

Changed the custom infiniteScroll function with the one provided by the SDK, which faster and more reliable
Disabled the page's cache, which increased the total number of scraped tweets by 20-40%

Feature:

Add useCheerio option to scrape tweets using cheerio crawler instead of puppeteer crawler, cheerio is more reliable, but it doesn't work when provided with login cookies

2023-01-19

Fix:

Updated scrolling to be more efficient and fixed an issue where the scrolling would quite early
Increased the default maxIdleTimeoutSecs to 60, to ensure that all tweets are scraped

Feature:

Add startUrl to output

2023-01-11

Fix:

Made scrolling slower to reduce the overload on the CPU
Add a check to stop scrolling if the desired number of tweets is reached
Increased the default maxRequestRetries to 6

2023-01-10

Feature:

Add option addTweetViewCount to include tweet view count (it's hidden and enabled by default)
Add view_count to the output if the option is enabled

2022-08-20

Feature:

Don't retry non-existing profiles
More efficient scrolling
Max concurrency increased to 3 by default

Fix:

Login modal blocking the scrolling
More resilient URL inputs and normalization

2022-08-10

Feature:

Revamp to Typescript and Crawlee

Fix:

Hanging timers on CPU overload

2022-02-25

Fix:

Timeline v2 object

2022-02-10

Feature:

Added '#sort_index' to the output
Updated README

2022-02-03

Fix:

Thread replies

2021-11-03

Fix:

Search results

2021-08-09

Features:

Update SDK 2

Bug fixes:

User shape object for some profiles

2021-07-18

Features:

Update to SDK 1.3.1

Changes:

Change default timeout values
Retiring of broken sessions
Deals with pinned tweets
Add debug log

Bug fixes:

Fix thread extraction

2021-06-12

Features:

Update to SDK 1.2.1

Fixes:

New GraphQL format

2021-05-03

Features:

Update to SDK 1.1.2
Recursive "People" search
Tweaks to wording in README and INPUT schema

Bug fixes:

Filter cookies that lead to never loading page / 401 error
Fetch data from GraphQl responses

2021-03-18

Features:

Update to SDK 1.0.2

Fixes:

Clicking on non-replies buttons

2021-02-26

Features:

Scrape replies of replies

Fixes:

Improve scraping stability

2021-02-04

Features:

Add topics
Add hashtags URLs
Optimize end of listings
Labels for outputScraperFunction for various scraper phases

Fixes:

Deduplication of tweets
Force retiring forever failing proxies

2021-01-19

Add mentions, symbols, URLs and hashtags to output
Add threads/status links support

2021-01-12

BREAKING CHANGE: Format of the dataset has changed
Search multiple terms at once, search hashtags and terms
Enriched user profile information (some information are only available when logged in)
Added minimum and max tweet dates
Updated SDK version
Custom data
Powerful extend output / scraper function

2020-11-25

Remove the need to provide credentials
Update SDK version
Allow to filter profile tweets for own tweets or include replies
Scrape faster when there's no login information
Accept twitter URLs, handles or @usernames for better user experience
Throws immediately if invalid handles are passed

Developer

Quacker

Actor metrics

1.2k monthly users
33.9% runs succeeded
7.1 days response time
Created in Jun 2019
Modified 9 days ago

Categories

Social media

News

Marketing

Google Maps Scraper

compass/crawler-google-places

Extract data from hundreds of Google Maps locations and businesses. Get Google Maps data including reviews, images, contact info, opening hours, location, popular times, prices & more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Compass

60.3k

Google Search Results Scraper

apify/google-search-scraper

Scrape Google Search Engine Results Pages (SERPs). Select the country or language and extract organic and paid results, ads, queries, People Also Ask, prices, reviews, like a Google SERP API. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Apify

42.8k

Instagram Hashtag Scraper

apify/instagram-hashtag-scraper

Scrape Instagram hashtags data. Just add one or more hashtags and extract posts, images, URLs, comments, likes, users, locations, timestamps, and more. Export scraped datasets, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

13.2k

Facebook Posts Scraper

apify/facebook-posts-scraper

Extract data from hundreds of Facebook posts from one or multiple Facebook pages and profiles. Get post URL, post text, page or profile URL, timestamp, number of likes, shares, comments, and more. Download the data in JSON, CSV, and Excel and use it in apps, spreadsheets, and reports.

Apify

7.1k

Contact Details Scraper

vdrmota/contact-info-scraper

Free email extractor to extract and download emails, phone numbers, Facebook, Twitter, LinkedIn, and Instagram profiles from any website. Extract contact information at scale from lists of URLs and download the data as Excel, CSV, JSON, HTML, and XML.

Vojta Drmota

16.8k

TikTok Data Extractor

clockworks/free-tiktok-scraper

Extract data about videos, users, and channels based on hashtags or scrape full user profiles including posts, total likes, name, nickname, numbers of comments, shares, followers, following, and more.

Clockworks

10.8k

Youtube Scraper

streamers/youtube-scraper

YouTube crawler and video scraper. Alternative YouTube API with no limits or quotas. Extract and download channel name, likes, number of views, and number of subscribers.

Streamers

3.2k

Instagram Scraper

apify/instagram-scraper

Scrape and download Instagram posts, profiles, places, hashtags, photos, and comments. Get data from Instagram using one or more Instagram URLs or search queries. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

36.5k

Facebook Ads Scraper

apify/facebook-ads-scraper

Extract advertising data from one or multiple Facebook Pages. Get page details, reach estimates, publisher platforms, report count, number of impressions, ad IDs, timestamps, and more. Download Facebook ads data in JSON, CSV, and Excel and use it in apps, spreadsheets, and reports.

Apify

3.8k

How to download tweets (no coding)

How to scrape Twitter (X.com) data using Python without Twitter API

Best web scraping tools in 2024

Build new tools

Are you a developer? Build your own Actors and run them on Apify.

Learn more

Get a custom solution

Get a custom web scraping or RPA solution.

Book a demo

2024-03-07

2024-01-04

2023-10-17

2023-08-10

2023-08-03

2023-07-12

2023-07-10

2023-06-30

2023-06-22

2023-06-19

2023-06-05

2023-05-30

2023-05-29

2023-05-09

2023-05-05

2023-04-28

2023-04-25

2023-04-09

2023-04-07

2023-04-05

2023-04-04

2023-04-03

2023-03-29

2023-03-15

2023-03-02

2023-02-02

2023-01-25

2023-01-23

2023-01-21

2023-01-19

2023-01-11

2023-01-10

2022-08-20

2022-08-10

2022-02-25

2022-02-10

2022-02-03

2021-11-03

2021-08-09

2021-07-18

2021-06-12

2021-05-03

2021-03-18

2021-02-26

2021-02-04

2021-01-19

2021-01-12

2020-11-25

You might also like these Actors

Google Maps Scraper

Google Search Results Scraper

Instagram Hashtag Scraper

Facebook Posts Scraper

Contact Details Scraper

TikTok Data Extractor

Youtube Scraper

Instagram Scraper

Facebook Ads Scraper

Related articles

Where next?

Build new tools

Get a custom solution

2024-03-07

2024-01-04

2023-10-17

2023-08-10

2023-08-03

2023-07-12

2023-07-10

2023-06-30

2023-06-22

2023-06-19

2023-06-05

2023-05-30

2023-05-29

2023-05-09

2023-05-05

2023-04-28

2023-04-25

2023-04-09