Pay $1.50 for 1,000 tweets
Twitter Scraper
Pay $1.50 for 1,000 tweets
Twitter Scraper searches and extracts data from Twitter (with historic data from 2006) - parses and converts the data to structured formats: HTML table, JSON, CSV, Excel and XML.
Best strategy for scraping entire history of one user?
Closed
I'm trying to scrape the entire history of one user. I understand, from reading previous issues, that I need to use a since/until search. When I do that I can see that it's not returning all of the tweets from a specific date. What is the best way to handle this?
Hi, can you please share with me any sample run?
jnr
Thanks for the quick response. I just realized that specifying "Latest" in the "Do you want to filter by content?" is not enough. The url has to include the "f=live" parameter to return all of the tweets for a specific day. Is that correct? So, for example, this run looks like it worked correctly: smcyF9eCmWk0TqoIb
If so, what is the best way to handle retrieving a month's worth of tweets? I understand that I'll get about max 800 results per run. So should I estimate how many days' worth of tweets this is, and then do a run for that amount? So, for the above, about one run per week of tweets?
Hi, you can use the search input option and filter by date ranges to be able to pull more results. For example, to pull results for the Twitter @apify from January to May, you can have the input as:
1{ 2 "addUserInfo": true, 3 "maxRequestRetries": 5, 4 "maxTweets": 100000, 5 "maxTweetsPerQuery": 1000, 6 "scrapeTweetReplies": true, 7 "searchMode": "live", 8 "searchTerms": [ 9 "(from:apify) until:2024-05-13 since:2024-05-01", 10 "(from:apify) until:2024-04-30 since:2024-04-01", 11 "(from:apify) until:2024-03-31 since:2024-03-01", 12 "(from:apify) until:2024-02-29 since:2024-02-01", 13 "(from:apify) until:2024-01-31 since:2024-01-01" 14 ] 15} 16 17See sample run: https://console.apify.com/view/runs/aPKkGU1LjzeEaV8NN 18 19Please let me know if you need further clarification on this.
Caleb David- 2.2k monthly users
- 100.0% runs succeeded
- 0.41 days response time
- Created in Jul 2023
- Modified 3 days ago
