Twitter Profile Scraper avatar

Twitter Profile Scraper

Try for free

3 days trial then $10.00/month - No credit card required now

View all Actors
Twitter Profile Scraper

Twitter Profile Scraper

epctex/twitter-profile-scraper
Try for free

3 days trial then $10.00/month - No credit card required now

Scrape any Twitter profile of any user. Extract tweets, replies, favorites, retweets, conversation threads and many more without any limit. Gather user related information such as verification, location, profile image, friends, followers, following and much more! No limits on Twitter!

SP

Does not work

Closed

spr123 opened this issue
10 months ago

Does not work

epctex avatar

epctex (epctex)

10 months ago

Hey there!

It seems like the actor is working properly and your data can be seen in the Storage Tab.

Please let us know if you need any further assistance, Best

SP

spr123

10 months ago

Its not working,

"There are no items in the dataset."

https://console.apify.com/view/runs/7vqRvcWWNQHqW9YT4

epctex avatar

epctex (epctex)

10 months ago

Hey again!

That is the Output Schema and it shows up empty only when you use "onlyUserInfo" as true. You can see the dataset by following the steps;

  • Click on Output
  • Click on "All fields"

This way, you can see all the output without any output schema. I am attaching a file for the reference.

Can you please let me know how it goes? Best

SP

spr123

10 months ago

Ok well, that is confusing, why you don't have the basic there is really strange.

I exported the full list but unfortunately it is no good. The full twitter url is not showing, so we have no way of matching up the data.

epctex avatar

epctex (epctex)

10 months ago

Hey,

This attribute is a new one. Therefore the output schema is focused on tweet retrieval. For more detailed information, please reach out to the Apify team.

The data you get is; raw data that we are getting directly from Twitter. Do you need the URL of the account you put in the Input Schema to match up the results? Or something different? Best

SP

spr123

10 months ago

"Do you need the URL of the account you put in the Input Schema to match up the results?"

Yes exactly, you have the outputted data, but without the url you inputted its pretty useless if its not matched up :)

Many thanks

Scott

epctex avatar

epctex (epctex)

10 months ago

Hey again!

Just deployed a new version for you which includes a new attribute called accountUrl. Please use the latest version for the best experience.

Can you please check this up and let us know how it goes? Best

SP

spr123

10 months ago

Looks great thanks

one last thing, we have a list of 7000 urls we want to do, I am 99% sure not all are working or correct urls, I tested your actor with a twitter url that does not exist but it seems to just keep going so I had to abort it,

https://console.apify.com/actors/runs/zon8sQIPCGy7zskA3#output

can you look into quickly flagging these and not charging too much and move onto the next url ?

epctex avatar

epctex (epctex)

10 months ago

Hey!

Unfortunately, that is not possible without opening the link. However, we just deployed a new version to quickly identify the non-existing user without retrying. I hope that helps.

Best

SP

spr123

10 months ago

Ok thanks, I ran the two urls and see you only show the valid one in the output.

Is it possible to list the bad one as well in the output ? and flag it as a non existent twitter profile

When you click on this one for example

https://twitter.com/fragrancexwdqwed

it says

This account doesn’t exist Try searching for another.

This would be a very good feature for people who add large amounts of twitter urls. Maybe flag it if the twitter profile has zero followers ? I will leave it with you to best flag the dead url.

Many thanks

Scott

SP

spr123

10 months ago

One other thing I have noticed, its quite slow due to the amount of data you are getting, even with "Only get the User Information" selected on.

On the output spreadsheet there are 80 fields ! If you can pick which fields you want, this would be great.

Many thanks

Scott

epctex avatar

epctex (epctex)

10 months ago

Hey again!

Sorry for the late reply. We just added a new feature called addNotFoundUsersToOutput. If you enable this option and onlyUserInfo at the same time, it will put the not found user in the output. Regarding the slowness, it is not actually because of the amount of data but mostly related to how Twitter behaves with the Datacenter IPs. Since the data is raw, there are not many things that can be done at that point.

However, if you still want to reduce the amount of data you are getting, you can customMapFunction where you can map the output data and retrieve the data you want in any way.

Please let us know if you need any further assistance, Best

3 months ago

Hi! I appreciate this tool and the way it's set up - unfortunately, I'm unable to get any results for when I try to pull tweets. I'm using a residential proxy, etc, but I'm not getting any results when I use any options other than just getting the user data. Has twitter blocked this scraper for getting individual tweets, or is there something I'm missing?

Here is my attempt to get individual posts: https://console.apify.com/view/runs/x2TqspiL7uShZBMDo

and here is the attempt to get just profile information (which also doesn't seem to have any information?)

https://console.apify.com/view/runs/2nWNhnxwgJghvY8EP

Thank you!

Developer
Maintained by Community

Actor Metrics

  • 104 monthly users

  • 15 stars

  • >99% runs succeeded

  • Created in Jul 2022

  • Modified 19 hours ago