Pay $2.50 for 1,000 posts
Twitter URL Scraper
Pay $2.50 for 1,000 posts
Copy any Twitter URL and extract Twitter usernames, profile photos, follower count, tweets, hashtags, favorite count, and more. Export scraped datasets, run the scraper via API, schedule and monitor runs or integrate with other tools.
All replies?
Closed
Does appear to scrape all replies to a tweet if the URL of that tweet is the input.
For example, tested with a tweet which the scraper's first result confirms has 1,690 replies, yet only 249 tweets appear in the result. How to scrape all of a single tweet's replies?
Hi B,
The tweet only has ~250 replies, and all of them are being scraped, the "reply_count": 1689 includes the replies for the replies and we don’t scrape those, we only scrape the visible ones, e.g. some of the replies in the image have their own replies Show replies:
bigchase
The other approach then is to use the Search Scraper to search the conversation (in this case, conversation_id:1631773371905241088). However, for some reason Twitter web search is not responding to conversation_id:##### search though my use of the API's search endpoints is responsive to queries containing conversation_id:#####. Any idea why?
In other words, this produces no results: https://twitter.com/search?q=conversation_id%3A1631773371905241088&src=typed_query
But this API endpoint does produce results for the same query: https://developer.twitter.com/en/docs/twitter-api/tweets/search/api-reference/get-tweets-search-recent
bigchase
Is there any other way to scrape a conversation-- the main thread and all replies to the replies (without having to gather the tweet ids of all tweets in the main thread and running this URL scraper on all of them individually)?
The first question is because we are not using the official API.
Second, currently, we don't have a way to do that using the scraper.
However, a workaround is after you scrape the tweets, you can scrape the url of the replies and you can scrape their replies.
Example: After scraping this tweet and its replies: https://twitter.com/LegionHoops/status/1631773371905241088 Go through the replies and scrape their replies: https://twitter.com/jasonmezrahi/status/1631853222431186949
The option to recursively scrape the whole tree of replies would be a possible addition, we will add it to the backlog.
bigchase
Thanks
Quacker- 327 monthly users
- 89.1% runs succeeded
- 12.1 days response time
- Created in Mar 2022
- Modified about 2 months ago
