Twitter Scraper

The date filters only apply when you are using the searchTerms option.

thanks for reply. is there any way for me to scrape a user's all tweets in last week?

@Caleb; even when using the searchTerms option the date filters aren't respected.

e.g. querying for tweets since yesterday and receiving tweets from months ago

Hi metal_jar, here's an example of scraping all tweets within 7 days from the user handle @apify.

1{
2  "addUserInfo": true,
3  "maxRequestRetries": 6,
4  "maxTweets": 5,
5  "scrapeTweetReplies": true,
6  "searchMode": "live",
7  "searchTerms": [
8    "(from:apify) until:2024-03-22 since:2024-03-14"
9  ]
10}
11
12See the example run: https://console.apify.com/view/runs/PUjYxInivC241swYh
13
14Hope that helps

Hi qback_au, can you please share with me a sample run?

@Caleb - See this run - date filters not working

https://console.apify.com/view/runs/VWpmkB4im5Tdwf6Ry

That is because there were no results for your provided search query (see image below). Since you provided the handle, the actor proceeded to pull data from that handle, this time without date filters. Please double-check Twitter UI to see if your queries yield any results before using it on the actor 🙂

Please double-check Twitter UI to see if your queries yield any results before using it on the actor 🙂

That defeats the point of scraping programmatically doesn't it?

Since you provided the handle, the actor proceeded to pull data from that handle, this time without date filters.

Can this unexpected behaviour be prevented through a different input? Just not providing the handle attribute?

That defeats the point of scraping programmatically doesn't it?

I mean that you should do a few tests search on the UI and ensure that it yields results before running the scrape. E.g., ensure that all the search filters used COULD potentially yield results before running it at scale/automatically on the actor

Can this unexpected behavior be prevented through a different input? Just not providing the handle attribute?

Yes, leaving that input blank will do the trick.

I hope that helps! 😅

@Caleb thanks a lot for your example config, I will try it out.

@Caleb I tried it and indeed can collect 20 pieces of data, that is to say, after I enabled the service, I can obtain all the posts within the specified time range ? There is also a question that needs to be consulted. I have set "scrapeTweetReplies": True here, but the returned result does not have the reply content. May I ask how to obtain the reply content?