Hello,

Our engineering team checked the runs and we couldn't find any issues on our end. Our scraper uses the query you give to it with Twitter and returns you whatever it can get from it. There is no additional logic, including removing duplicate. It never alters the output.

Twitter changes behaviour constantly and it is better to test your queries on twitter web UI before running the actor. Another reason can be using multiple from queries withORs. Are you sure that works with Twitter? If you are using ORs, I think the best approach would be to create separate runs for each profile.

Does that make sense?

Cheers!

I tried agian with only one from, but there is many duplicate as well。 https://console.apify.com/actors/61RPP7dywgiy0JPD0/runs/8tXLNNdSegCRGCYue

https://console.apify.com/actors/61RPP7dywgiy0JPD0/runs/kShmSwwrGcfhDee0V this run duplicate as well

results are duplicated 27 times for my last run.

Hey hey,

As I mentioned, this is something we have no control over. Our actor uses Twitter search and paginates as long as it can get the pagination. And when you try to fetch lots of tweets, this is inevitable since Twitter acts very weird with long paginations.

For fetching a profile, I suggest you to use a similar approach as we explained here:

{
    "includeSearchTerms": false,
    "onlyImage": false,
    "onlyQuote": false,
    "onlyTwitterBlue": false,
    "onlyVerifiedUsers": false,
    "onlyVideo": false,
    "searchTerms": [
        "from:NASA since:2023-01-01 until:2023-02-01",
        "from:NASA since:2023-02-01 until:2023-03-01",
        "from:NASA since:2023-03-01 until:2023-04-01",
        "from:NASA since:2023-04-01 until:2023-05-01",
        "from:NASA since:2023-05-01 until:2023-06-01",
        "from:NASA since:2023-06-01 until:2023-07-01",
        "from:NASA since:2023-07-01 until:2023-08-01",
        "from:NASA since:2023-08-01 until:2023-09-01",
        "from:NASA since:2023-09-01 until:2023-10-01",
        "from:NASA since:2023-10-01 until:2023-11-01",
        "from:NASA since:2023-11-01 until:2023-12-01"
    ],
    "sort": "Latest",
    "tweetLanguage": "en"
}

That way, you will have less paginations and your results will have less duplicates.

Cheers!

https://console.apify.com/actors/61RPP7dywgiy0JPD0/runs/sJdAPAajvSfCpQpud

I've tried this way.. but when there is no reuslt return. it still show 33 outputs.