.actor

FROM apify/actor-python:3.11

COPY requirements.txt ./

RUN echo "Python version:" \
 && python --version \
 && echo "Pip version:" \
 && pip --version \
 && echo "Installing dependencies:" \
 && pip install -r requirements.txt \
 && echo "All installed Python packages:" \
 && pip freeze

COPY . ./

CMD ["python3", "-m", "src.__main__"]


.actor/Dockerfile

Twitter Scraper

.actor/actor.json

Set an overall limit for the number of tweets to scrape across all threads. Once this limit is reached, all threads will stop. This may lead to unequal numbers of tweets collected from each query. To ensure equal distribution, consider using the 'Max Tweets Per Input' setting.

Overall Maximum Tweets

Filters content to only return tweets from verified users with Twitter Blue

Only verified users with Twitter Blue

Has some engagement (replies, favorites, retweets). Can be negated to find tweets with no engagement.

Has some engagement

Filters content to only return tweets with any image

All images

Filters content to only return tweets with any media type

All media types

Only retweets created using the retweet button. Works well combined with from: to show only retweets. Only works within the last 7-10 days or so.

Only retweets made using the retweet button

Filters content to only return Quote Tweets

Contain Quote Tweets

Returns tweets that are a reply to another Tweet

Only replies

Old style retweets ('RT') + quoted tweets

Only retweets or quoted tweets

Only safe tweets

Filters content to only return tweets with native Twitter images (pic.twitter.com links)

Native Twitter images

Filters content to only return tweets from verified users

Only verified users

Filters content to only return tweets with any video type, including native Twitter video and external sources such as Youtube.

All video types

Written by a particular user. Useful if you want to use multiple custom queries for a single user.'

Author

Provide a list of Twitter profiles to scrape. The profiles should be written as 'username' or '@username'

Options to scrape twitter through specific profile's to look at that user's replies and tweets. Works with filters.

Specific Twitter Profiles

Geotagged in this place. Also supports Phrases, eg: near:'The Hague'

Geotagged near

To get tweets around a specific location, use geocode:lat,long,radius. E.g., to get tweets 10km around twitters hq, use geocode:37.7764685,-122.4172004,10km

Geocode

Choose a specific language to filter tweets. If left as 'Any', tweets in all languages will be included.

Tweet Language

Specify the maximum number of attempts to be made if a scrape operation fails. After reaching this limit, the failed operation will be skipped.

Options to customize the run through changing the maximum number of attempts and threads.

Maximum Attempts

Set the maximum number of tweets to scrape per query, profile, or URL. For instance, if you set a maximum of 50 for 3 search terms, the total tweet limit will be 150.

Max Tweets Per Query/Profile/URL

Mentioning

Specify minimum number of favorites a tweet should have to be included.

Minimum Favorites

Specify minimum number of replies a tweet should have to be included.

Minimum Replies

Specify minimum number of Retweets a tweet should have to be included.

Minimum Retweets

Specify the earliest date (inclusive) from which to start scraping tweets. The date should be in the 'YYYY-MM-DD' format.

To set a custom time frame for tweets published between start inclusive and end exclusive, (from start up to be not including end).

Start

Overrides the number of threads/workers automatically optimized for your memory amount. Could be useful for troubleshooting.

Manually Set Number of Threads/Workers (if blank it will automatically optimize based on memory)

Specify the latest date (exclusive) until which to scrape tweets. The date should be in the 'YYYY-MM-DD' format.

Filters content to only return tweets that are not retweets, replies, or quotes.

Various options to filter content for ease of use. Feel free to just use queries instead, in which case see https://github.com/igorbrigadir/twitter-advanced-search for more details

Only tweets (no retweets, replies or quotes)

Search tweets by Place Object ID eg: USA Place ID is 96683cc9126741d1

Place Object ID

Enter your search queries here. For example search queries, refer to https://github.com/igorbrigadir/twitter-advanced-search.

Search Terms

List the full links to tweets you want to scape such as 'https://twitter.com/search?q=((MadMaxFuryRoad%20OR%20MadMax%20OR%20%22FuryRoad%22%20OR%20%22GeorgeMiller%22%20OR%20GeorgeMiller%20OR%20TomHardy%20OR%20%22CharlizeTheron%22%20OR%20CharlizeTheron)%20AND%20(Film%20OR%20Movie%20OR%20Review))&src=typed_query&f=top' if your url contains more such as a cursor only the tweets on the current screen will be loaded.

Twitter Search URLs

In reply to

List the twitter search URLs you want to scape such as 'https://twitter.com/elonmusk/status/1656898688739057665' if your url contains more such as a cursor like 'https://twitter.com/elonmusk/status/1656898688739057665?s=20' only the tweets on the current screen will be loaded.

Options to scrape twitter through URLs to look at specific tweets. Currently supports scraping information from a tweet URL or a search URL. Does not work with filters.

Specific Tweet URLs

Activates an experimental profile scraper that uses the user timeline instead of search and should have a few months of history (can go back a few months rather than 10 days) but doesn't work with filtering, except tweet only will only return the user's tweets and replies.

Use Experimental Profile Scraper

Options to either collect each tweet with all of its replies, only each tweet, or only the replies.

Info to collect

Within specific radius of the 'near' operator, to apply a limit. Can use km or mi. e.g. fire near:san-francisco within:10km

Within Radius

Tweet-Flash Input Schema

.actor/input_schema.json

The total number of followers the scraped user has

The total number of users the scraped scraped user is following

The total number of likes the scraped user has from all posts

The total number of tweets by the scraped user

A list of the hashtags contained in a given tweet

A list of the links contained in a given tweet. 

A list of the mentions contained in a given tweet

Overview

.actor/output_schema.json

""" Copyright (C) Jonathan Larson - All Rights Reserved
 Unauthorized copying of this file, via any medium is strictly prohibited
 Proprietary and confidential
 Written by Jonathan Larson <joanthanlarsonbusiness@gmail.com>, May 2023
 """

src/__init__.py

""" Copyright (C) Jonathan Larson - All Rights Reserved
 Unauthorized copying of this file, via any medium is strictly prohibited
 Proprietary and confidential
 Written by Jonathan Larson <joanthanlarsonbusiness@gmail.com>, May 2023
 """

import asyncio
import logging

from apify.log import ActorLogFormatter

from .main import main

handler = logging.StreamHandler()
handler.setFormatter(ActorLogFormatter())

apify_client_logger = logging.getLogger('apify_client')
apify_client_logger.setLevel(logging.INFO)
apify_client_logger.addHandler(handler)

apify_logger = logging.getLogger('apify')
apify_logger.setLevel(logging.DEBUG)
apify_logger.addHandler(handler)

if __name__ == "__main__":
    asyncio.run(main())


src/__main__.py

src/main.py

""" Copyright (C) Jonathan Larson - All Rights Reserved
 Unauthorized copying of this file, via any medium is strictly prohibited
 Proprietary and confidential
 Written by Jonathan Larson <joanthanlarsonbusiness@gmail.com>, May 2023
 """

class Tweet:
    def __init__(self, tweet_avatar, tweet_id, url, text, username, fullname, timestamp, language, images, in_reply_to, user_info, stats, query, tweet_links, tweet_hashtags, tweet_mentions):
        self.tweet_avatar = tweet_avatar
        self.tweet_id = tweet_id
        self.url = url
        self.text = text
        self.username = username
        self.fullname = fullname
        self.timestamp = timestamp
        self.language = language
        self.images = images
        self.in_reply_to = in_reply_to
        self.user_info = user_info
        self.stats = stats
        self.query = query 
        self.tweet_links = tweet_links
        self.tweet_hashtags = tweet_hashtags
        self.tweet_mentions = tweet_mentions


class TweetStats:
    def __init__(self, replies, retweets, quotes, likes):
        self.replies = replies
        self.retweets = retweets
        self.quotes = quotes
        self.likes = likes

class UsersScanned:
 def __init__(self, username, banner_image, total_tweets, num_following, num_followers, total_likes):
    self.username = username
    self.banner_image = banner_image
    self.total_tweets = total_tweets
    self.num_following = num_following
    self.num_followers = num_followers
    self.total_likes = total_likes

src/models.py

""" Copyright (C) Jonathan Larson - All Rights Reserved
 Unauthorized copying of this file, via any medium is strictly prohibited
 Proprietary and confidential
 Written by Jonathan Larson <joanthanlarsonbusiness@gmail.com>, May 2023
 """

def init():
    global max_workers
    global workers_used 
    global total_users_scanned
    global language_input
    global detect_language
    global num_pushed
    global max_tweets

src/settings.py

src/utils.py

""" Copyright (C) Jonathan Larson - All Rights Reserved
 Unauthorized copying of this file, via any medium is strictly prohibited
 Proprietary and confidential
 Written by Jonathan Larson <joanthanlarsonbusiness@gmail.com>, May 2023
 """

# configurations
.idea

# installed files
.venv

# git folder
.git

# temporary files and caches
__pycache__
*.pyc
*.pyo
*.pyd
*.pyc
*.log
*.egg-info/
dist/
build/


.dockerignore

# Copyright (C) Jonathan Larson - All Rights Reserved 
# Unauthorized copying of this file, via any medium is strictly prohibited 
# Proprietary and confidential
# Written by Jonathan Larson <joanthanlarsonbusiness@gmail.com>, May 2023

root = true

[*]
indent_style = space
indent_size = 4
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true
end_of_line = lf


.editorconfig

# Copyright (C) Jonathan Larson - All Rights Reserved 
# Unauthorized copying of this file, via any medium is strictly prohibited 
# Proprietary and confidential
# Written by Jonathan Larson <joanthanlarsonbusiness@gmail.com>, May 2023

.idea
.DS_Store
.venv/
.env/
__pypackages__
dist/
build/
*.egg-info/
*.egg
__pycache__
.mypy_cache
.dmypy.json
dmypy.json
.pytest_cache
*.log


.gitignore

Has some engagement (replies, likes, retweets). Can be negated to find tweets with no engagement.

Various options to filter content, see https://github.com/igorbrigadir/twitter-advanced-search for more details

List the Twitter profiles you want to scrape (ex: elonmusk or @elonmusk), which will be outputted separately

Scrape certain Twitter profiles

Language

Specify maximum number of attempts to try.

Specify maximum number of tweets to scrape.

Maximum Tweets

Specify minimum number of Likes a tweet should have to be included.

Minimum Likes

Newer than

Manually Set Number of Threads/Workers (0 automatically optimizes based on memory)

To set a custom time frame for tweets starting at newer than and going up to older than.

Older than

Queries

Twitter Scraper Input Schema

idk.json

# Copyright (C) Jonathan Larson - All Rights Reserved
# Unauthorized copying of this file, via any medium is strictly prohibited
# Proprietary and confidential
# Written by Jonathan Larson <joanthanlarsonbusiness@gmail.com>, May 2023


# Add your dependencies here.
# See https://pip.pypa.io/en/latest/reference/requirements-file-format/
# for how to format them
#apify ~= 1.0.0
apify==1.0.0
scrapy
selenium
beautifulsoup4==4.12.2
regex==2023.3.23
Requests==2.30.0
lxml
langdetect
python-dateutil
asyncio 
aiohttp
psutil


requirements.txt

Search, Filter, URL, User: This All-In-One Twitter Scraper delivers tweets at Blazing Fast Speeds with Historical Search and Other Unique Features. Search is no longer limited by date! 
 
It's fixed after facing issues for the past week, but please comment under issues if you face any while using.

tweet-flash

Tweet Flash - Twitter Scraper (Working Again as of 9/21)

Tweet Flash - Twitter Scraper (Working Again as of 9/21) · Apify

Input · Tweet Flash - Twitter Scraper (Working Again as of 9/21) · Apify

API · Tweet Flash - Twitter Scraper (Working Again as of 9/21) · Apify

Related Actors · Tweet Flash - Twitter Scraper (Working Again as of 9/21) · Apify

Search, Filter, URL, User: This All-In-One Twitter Scraper delivers tweets at Blazing Fast Speeds with Historical Search and Other Unique Features. Search is no longer limited by date! 
 
It's fixe...

Where next?