🏯 Tweet Scraper V2 (Pay Per Result) - X / Twitter Scraper

Might this be related to the very large time span which our query covers? I noted your response to another user concerning breaking the query dates into smaller segment.

I went ahead and did month-by-month and week-by-week searches as per the below screenshots, but these produced even fewer results, Whole time range: 66 of 70 Month-by-Month: 50 of 70 Week-by-week: 44 of 70

Example of actor inputs:

{'customMapFunction': '(object) => { return {...object} }', 'includeSearchTerms': True, 'maxItems': 23810, 'maxTweetsPerQuery': 23810, 'onlyImage': False, 'onlyQuote': False, 'onlyTwitterBlue': False, 'onlyVerifiedUsers': False, 'onlyVideo': False, 'searchTerms': ['(ชายแดนใต้ OR ไฟใต้) (ไทยพุทธ) since:2022-01-01 until:2022-01-07\t', '(ชายแดนใต้ OR ไฟใต้) (ไทยพุทธ) since:2022-01-08 until:2022-01-14\t', '(ชายแดนใต้ OR ไฟใต้) (ไทยพุทธ) since:2022-01-15 until:2022-01-21\t', '(ชายแดนใต้ OR ไฟใต้) (ไทยพุทธ) since:2022-01-22 until:2022-01-28\t', ...], 'sort': 'Latest'}

Hey hey,

Can you send us how you get 70 results from Twitter so we can investigate further?

Best

Sure: I typed the original query into X and counted as I knew prior to this that this topic could be used for testing. :) (Literally (ชายแดนใต้ OR ไฟใต้) (ไทยพุทธ) since:2022-01-15 until:2022-01-21)

I know it isn't exactly the most programatic solution, but it works for test sets.

I can also indicate the missing four posts if this is of any assistance?

Hey hey,

Yes that would be great. Can you also send me your direct search URL? When I search these, I cannot get any results.

Cheers

Search Results: https://x.com/search?q=%E0%B9%84%E0%B8%97%E0%B8%A2%E0%B8%9E%E0%B8%B8%E0%B8%97%E0%B8%98%20(%E0%B8%8A%E0%B8%B2%E0%B8%A2%E0%B9%81%E0%B8%94%E0%B8%99%E0%B9%83%E0%B8%95%E0%B9%89%20OR%20%20%E0%B9%84%E0%B8%9F%E0%B9%83%E0%B8%95%E0%B9%89)%20until%3A2024-07-24%20since%3A2022-01-01&src=typed_query&f=live

1. The post with the photo of the military person: 2nd in the list of returned tweets 2 & 3 - Middle Section of the results
2. Very far towards the end.

Also, I note that there is a difference between how the search terms are ordered and we also experimented with this, both versions below produce the same results (66).

Version 1: ไทยพุทธ (ชายแดนใต้ OR ไฟใต้) until:2024-07-24 since:2022-01-01 Version 2: (ชายแดนใต้ OR ไฟใต้) ไทยพุทธ until:2024-07-24 since:2022-01-01

Although the order of the results did vary.

Hello,

We are checking the missing tweet issue. For the other one, that is an expected behavior since Twitter searches by keyword and it doesn't check the order. You can try to put these keywords between quotes if you want to get the exact result like "ไทยพุทธ (ชายแดนใต้ OR ไฟใต้) ". I believe that is the suggested way of doing a search like this. You can refer to https://github.com/igorbrigadir/twitter-advanced-search to get more information

Cheers

Thanks for prioritizing the missing tweets. I am not too concerned with the order of results, as that is of no consequence. Although I question whether wrapping the boolean operators in double quotes is recommended? That is used for exact phrase matches, isn't it? Wouldn't that effectively render the implicit 'AND' and 'OR' as keywords?

Hey there,

Just got a reply from our engineering team. They investigated the missing tweet however couldn't find the real cause here. One of the possible reasons is that these tweets could be marked as sensitive. Our scrapers returning you whatever they can get from Twitter so sometimes there can be some inconsistencies or missing data. We cannot promise anything related to the data we get from Twitter.

For the Twitter queries, I think that github link has the most information out there. We don't have any more information on top of that.

I hope these will help.

Cheers!