Twitter Scraper avatar
Twitter Scraper
Try for free

Pay $1.50 for 1,000 tweets

View all Actors
Twitter Scraper

Twitter Scraper

microworlds/twitter-scraper
Try for free

Pay $1.50 for 1,000 tweets

Twitter Scraper searches and extracts data from Twitter (with historic data from 2006) - parses and converts the data to structured formats: HTML table, JSON, CSV, Excel and XML.

User avatar

the time range function doesn't work

Closed

metal_jar opened this issue
a month ago

I want to scrape a user's tweets of last month and I set the sinceDate and untilDate values, but the items got in result is still the latest date.

User avatar

The date filters only apply when you are using the searchTerms option.

User avatar

metal_jar

a month ago

thanks for reply. is there any way for me to scrape a user's all tweets in last week?

User avatar

qback_au

a month ago

@Caleb; even when using the searchTerms option the date filters aren't respected.

e.g. querying for tweets since yesterday and receiving tweets from months ago

User avatar

Hi metal_jar, here's an example of scraping all tweets within 7 days from the user handle @apify.

1{
2  "addUserInfo": true,
3  "maxRequestRetries": 6,
4  "maxTweets": 5,
5  "scrapeTweetReplies": true,
6  "searchMode": "live",
7  "searchTerms": [
8    "(from:apify) until:2024-03-22 since:2024-03-14"
9  ]
10}
11
12See the example run: https://console.apify.com/view/runs/PUjYxInivC241swYh
13
14Hope that helps
User avatar

Hi qback_au, can you please share with me a sample run?

User avatar

qback_au

25 days ago

@Caleb - See this run - date filters not working

https://console.apify.com/view/runs/VWpmkB4im5Tdwf6Ry

User avatar

That is because there were no results for your provided search query (see image below). Since you provided the handle, the actor proceeded to pull data from that handle, this time without date filters. Please double-check Twitter UI to see if your queries yield any results before using it on the actor 🙂

User avatar

qback_au

25 days ago

Please double-check Twitter UI to see if your queries yield any results before using it on the actor 🙂

That defeats the point of scraping programmatically doesn't it?

Since you provided the handle, the actor proceeded to pull data from that handle, this time without date filters.

Can this unexpected behaviour be prevented through a different input? Just not providing the handle attribute?

User avatar

That defeats the point of scraping programmatically doesn't it?

I mean that you should do a few tests search on the UI and ensure that it yields results before running the scrape. E.g., ensure that all the search filters used COULD potentially yield results before running it at scale/automatically on the actor

Can this unexpected behavior be prevented through a different input? Just not providing the handle attribute?

Yes, leaving that input blank will do the trick.

I hope that helps! 😅

User avatar

metal_jar

16 days ago

@Caleb thanks a lot for your example config, I will try it out.

User avatar

metal_jar

16 days ago

@Caleb I tried it and indeed can collect 20 pieces of data, that is to say, after I enabled the service, I can obtain all the posts within the specified time range ? There is also a question that needs to be consulted. I have set "scrapeTweetReplies": True here, but the returned result does not have the reply content. May I ask how to obtain the reply content?

User avatar

Hi metal_jar,

I can obtain all the posts within the specified time range ?

Yes, that is correct.

May I ask how to obtain the reply content?

The scrapeTweetReplies option only works when you're scraping a tweet URL e.g., https://twitter.com/Tesla/status/177523894550621000, not on searches or tweet handles

Developer
Maintained by Community
Actor metrics
  • 4.5k monthly users
  • 97.0% runs succeeded
  • 0.4 days response time
  • Created in Jul 2023
  • Modified 22 days ago
Categories