Twitter/X Scraper (Python)

A powerful Apify actor to scrape Twitter/X data without requiring authentication. Uses Playwright for browser automation and extracts data directly from the Twitter/X website.

Features

✅ No Authentication Required - Works without login cookies
✅ Search Tweets - Find tweets by keywords, hashtags
✅ User Timelines - Get tweets from any public user
✅ User Profiles - Extract profile information
✅ Media & Replies - Filter for specific content types
✅ Engagement Metrics - Likes, retweets, replies, views
✅ Anti-Detection - Built-in stealth features

How It Works

1. Playwright Browser - Uses a real browser to access Twitter
2. Guest Token - Obtains guest credentials automatically
3. DOM Extraction - Parses tweets directly from the page
4. Infinite Scroll - Automatically scrolls to load more tweets

Input Parameters

Search Configuration

Parameter	Type	Description
search_terms	Array	Search queries (e.g., #AI, bitcoin)
usernames	Array	Twitter usernames to scrape (without @)
tweet_urls	Array	Direct URLs to specific tweets

Scraping Options

Parameter	Type	Default	Description
scrape_type	String	tweets	What to scrape: tweets, profile, replies, media
max_tweets	Integer	50	Maximum tweets per search/user
include_replies	Boolean	false	Include reply tweets
include_retweets	Boolean	true	Include retweets

Filters

Parameter	Type	Description
min_likes	Integer	Minimum likes filter
min_retweets	Integer	Minimum retweets filter

Output Format

Tweet Data

{
    "tweet_id": "1234567890",
    "text": "Tweet content here...",
    "author_name": "Elon Musk",
    "author_username": "elonmusk",
    "likes": 50000,
    "retweets": 10000,
    "replies": 5000,
    "views": 1000000,
    "created_at": "2024-01-15T10:30:00.000Z",
    "url": "https://x.com/elonmusk/status/1234567890",
    "is_retweet": false,
    "is_reply": false,
    "media": [{"type": "image", "url": "..."}],
    "hashtags": ["#tech"],
    "mentions": ["@OpenAI"],
    "scraped_at": "2024-01-15T12:00:00.000Z"
}

Profile Data

{
    "name": "Elon Musk",
    "username": "elonmusk",
    "bio": "...",
    "location": "Austin, TX",
    "website": "tesla.com",
    "join_date": "Joined June 2009",
    "followers": "170M",
    "following": "500",
    "is_verified": true,
    "profile_image_url": "...",
    "url": "https://x.com/elonmusk",
    "scraped_at": "2024-01-15T12:00:00.000Z"
}

Local Development

Prerequisites

• Python 3.9+
• pip

Setup

# Navigate to the actor directory
cd twitter-scraper-python

# Create virtual environment
python -m venv venv

# Activate (Windows)
.\venv\Scripts\activate

# Activate (Linux/Mac)
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Install Playwright browsers
playwright install chromium

Run Locally

# Using Apify CLI
apify run

# Or directly
python -m src.main

Test Input

Edit apify_storage/key_value_stores/default/INPUT.json:

{
    "usernames": ["elonmusk"],
    "max_tweets": 10
}

Deployment to Apify

# Login to Apify
apify login

# Push to Apify
apify push

Limitations

• Rate Limits: Twitter may rate-limit after many requests
• Login Walls: Some content requires authentication
• Dynamic Changes: Twitter's DOM may change

Tips

• Use residential proxies for better success rates
• Start with small limits to test
• Use Apify Proxy when deploying to Apify platform

License

ISC License