X (Twitter) Twitter Scraper - Smart Routing

A comprehensive Twitter scraper powered by X (Twitter) instances with smart automatic routing. No mode selection needed - just provide your inputs and the Actor automatically detects what to scrape! Includes full residential proxy support for reliable data extraction.

🚀 Features

Smart Automatic Routing

The Actor automatically detects what you want to scrape based on your inputs:

• Twitter Usernames → Automatically scrapes profiles (smart mode detection)
• Twitter List URLs → Automatically scrapes list content
• Search Terms → Automatically searches tweets or users

Intelligent Profile Scraping

When scraping profiles, the Actor automatically selects the best mode:

• ✅ Media Only Mode - Check Media Only to extract only tweets with images/videos
• ✅ With Replies Mode - Enable Include Replies to get all tweets including replies
• ✅ Normal Mode - Standard tweet scraping (default)
• ✅ Profile Search - Add search terms to find specific content within profiles

Advanced Features

• 🔒 Residential Proxy Support - Use Apify residential proxies for reliable scraping
• 🎯 Advanced Filtering - Filter by date, location, media type, engagement metrics
• 👤 Profile Information - Extract complete profile details (bio, stats, avatar, banner)
• 📷 Media Extraction - Get direct URLs for images and videos with HLS preview links
• 📊 Engagement Stats - Collect likes, retweets, comments, and quotes
• 🔄 Rate Limiting Protection - Built-in retry logic and exponential backoff
• 📄 Pagination - Automatic handling of multiple pages
• 📋 Structured Output - Clean JSON output to Apify dataset

📋 Input Configuration

Main Input Fields

1. Twitter Usernames

Add one or more Twitter usernames to scrape profiles. The Actor will automatically:

• Choose between normal, with-replies, or media-only mode based on your settings
• Apply profile search if search terms are provided

Examples:

elonmusk
apify
@naval (@ symbol is optional)

2. Twitter List URLs

Add Twitter/X list URLs to scrape all tweets from lists.

Supported formats:

https://x.com/i/lists/123456
https://twitter.com/i/lists/123456
https://X (Twitter).net/i/lists/123456

3. Search Terms

Add search queries to find tweets or users across Twitter.

Examples:

web scraping
AI technology
from:apify
#python

Profile Options

• Include Replies - Include reply tweets (automatically switches to "with replies" mode)
• Include Retweets - Include retweets in results
• Media Only - Only scrape tweets containing images or videos (uses dedicated media scraper)

Search Options

When using Search Terms, choose what to search:

• Search Tweets Globally - Find tweets across all Twitter
• Search Users - Find Twitter accounts
• Search Within Profiles - Search for specific content in provided profiles (requires Twitter Usernames)

Content Filters

Filter tweets by content characteristics:

• Only Images - Filter tweets containing images
• Only Videos - Filter tweets containing videos
• Only Verified Users - Only include verified accounts
• Only Quote Tweets - Only include quote tweets
• Include Native Retweets - Include native retweets
• Only Tweets with Links - Filter tweets with external links
• Safe Search - Enable to filter sensitive content

Date & Location Filters

• Start Date (From) - Filter tweets from this date onwards (YYYY-MM-DD format)
• End Date (Until) - Filter tweets up to this date
• Location Filter - Filter by location (e.g., "Los Angeles", "New York")

Engagement Filters

Set minimum engagement thresholds:

• Minimum Likes - Minimum number of likes required
• Minimum Replies - Minimum number of replies required
• Minimum Retweets - Minimum number of retweets required

Limits & Pagination

• Max Items to Scrape - Maximum items (tweets/users) to collect (0 = unlimited)
• Max Pages - Maximum pages to scrape per handle/query

Advanced Options

• Tweet Language - Filter by language code (e.g., "en", "es", "fr")
• Scrape Profile Info - Extract detailed profile information
• X (Twitter) Instance URL - Custom X (Twitter) instance (advanced users)

Proxy Settings

• Use Residential Proxy - Enable Apify residential proxy (requires paid plan)
• Proxy Configuration - Configure proxy groups and countries
• Max Retries per Request - Number of retry attempts for failed requests

🔧 Usage Examples

Example 1: Simple Profile Scraping

{
  "twitterHandles": ["elonmusk", "apify"],
  "maxItems": 100
}

What happens: Scrapes the last 100 tweets from each profile using normal mode.

Example 2: Profile with Replies

{
  "twitterHandles": ["elonmusk"],
  "includeReplies": true,
  "maxItems": 200
}

What happens: Automatically switches to "with replies" mode to get all tweets including replies.

Example 3: Media Only Scraping

{
  "twitterHandles": ["apify", "elonmusk"],
  "mediaOnly": true
}

What happens: Uses the dedicated media scraper to extract only tweets with images/videos.

Example 4: Global Tweet Search with Filters

{
  "searchTerms": ["web scraping", "#ai"],
  "searchType": "tweets",
  "startDate": "2024-01-01",
  "endDate": "2024-12-31",
  "onlyVerified": true,
  "minLikes": 10,
  "tweetLanguage": "en"
}

What happens: Searches for tweets globally with engagement and verification filters.

Example 5: Profile Search (Find Specific Content)

{
  "twitterHandles": ["apify"],
  "searchTerms": ["scraping", "automation"],
  "searchType": "profile_tweets",
  "startDate": "2024-01-01"
}

What happens: Searches for specific keywords within the provided profile's tweets.

Example 6: Scrape Twitter List

{
  "listUrls": ["https://x.com/i/lists/78783491"],
  "maxItems": 200
}

What happens: Scrapes up to 200 tweets from the specified list.

Example 7: User Search

{
  "searchTerms": ["data scientist", "ML engineer"],
  "searchType": "users",
  "maxItems": 50
}

What happens: Finds Twitter users matching the search terms.

Example 8: Combined Scraping

{
  "twitterHandles": ["apify"],
  "listUrls": ["https://x.com/i/lists/123456"],
  "searchTerms": ["web scraping"],
  "searchType": "tweets",
  "maxItems": 100
}

What happens: Scrapes all three: profile tweets, list tweets, and search results.

📊 Output Format

Data is saved to the Apify dataset with structured JSON objects:

Profile Output

{
  "type": "profile",
  "username": "apify",
  "fullname": "Apify",
  "bio": "Web scraping and automation platform...",
  "stats": {
    "tweets": "1,234",
    "followers": "12.5K",
    "following": "567"
  },
  "avatar": "https://pbs.twimg.com/...",
  "banner": "https://pbs.twimg.com/...",
  "website": "https://apify.com",
  "location": "San Francisco, CA",
  "joined": "Joined January 2024",
  "scrape_date": "2024-10-24T12:00:00.000Z"
}

Tweet Output

{
  "type": "tweet",
  "tweet_url": "https://x.com/apify/status/123456",
  "tweet_id": "123456",
  "profile_username": "apify",
  "author": {
    "fullname": "Apify",
    "username": "@apify",
    "avatar": "https://pbs.twimg.com/...",
    "verified": true,
    "verified_type": "blue"
  },
  "text": "Tweet content here...",
  "date": "Oct 24, 2024 · 12:00 PM UTC",
  "date_short": "Oct 24",
  "hashtags": ["#webscraping", "#automation"],
  "mentions": ["@user1", "@user2"],
  "urls": ["https://example.com"],
  "stats": {
    "comments": 12,
    "retweets": 34,
    "quotes": 5,
    "likes": 156
  },
  "media": [
    {
      "type": "video",
      "url": "https://video.twimg.com/...",
      "hls_preview": "https://hlsjs.video-dev.org/demo/?src=...",
      "thumbnail": "https://pbs.twimg.com/...",
      "requires_hls": false
    },
    {
      "type": "image",
      "url": "https://pbs.twimg.com/...",
      "thumbnail": "https://pbs.twimg.com/..."
    }
  ],
  "is_retweet": false,
  "scrape_date": "2024-10-24T12:00:00.000Z"
}

Media Tweet Output

{
  "type": "media_tweet",
  "profile_username": "apify",
  "tweet_id": "123456",
  "media": [
    {
      "type": "image",
      "url": "https://pbs.twimg.com/...",
      "thumbnail": "https://pbs.twimg.com/..."
    }
  ],
  "stats": {
    "comments": 5,
    "retweets": 10,
    "quotes": 2,
    "likes": 45
  },
  "scrape_date": "2024-10-24T12:00:00.000Z"
}

User Search Output

{
  "type": "user",
  "search_query": "data scientist",
  "username": "@john_doe",
  "fullname": "John Doe",
  "display_name": "John Doe",
  "handle": "@john_doe",
  "bio": "Data Scientist | ML Engineer...",
  "avatar": "https://pbs.twimg.com/...",
  "verified": true,
  "verified_type": "blue",
  "protected": false,
  "hashtags": ["#DataScience", "#ML"],
  "mentions": [],
  "urls": ["https://johndoe.com"],
  "profile_url": "https://x.com/john_doe",
  "scrape_date": "2024-10-24T12:00:00.000Z"
}

🛠️ Technical Details

Architecture

The Actor uses:

• Smart Routing Logic - Automatically detects scraping mode based on inputs
• X (Twitter) - Privacy-focused Twitter frontend for scraping
• BeautifulSoup - HTML parsing and data extraction
• httpx - Async HTTP client with HTTP/2 support
• Apify SDK - Full integration with Apify platform

Automatic Mode Detection

The Actor intelligently routes requests:

IF twitterHandles PROVIDED:
  IF mediaOnly = true → profile_media.py
  ELSE IF includeReplies OR includeRetweets → profile_with_reply.py
  ELSE → profile_main.py
  
  IF searchTerms PROVIDED AND searchType = "profile_tweets" → profile_search.py

IF listUrls PROVIDED → list_extractor.py

IF searchTerms PROVIDED (without twitterHandles):
  IF searchType = "tweets" → search_post.py
  ELSE IF searchType = "users" → search_user.py

Rate Limiting

• Automatic retry logic with exponential backoff
• Configurable delays between requests (4-8 seconds)
• Maximum retry attempts (default: 5)
• HTTP/2 connection pooling for efficiency

Proxy Support

• Full support for Apify residential proxies
• Automatic proxy rotation
• Configurable proxy groups and countries
• Fallback to non-proxy mode if configuration fails

📝 Notes

X (Twitter) Instances

The default X (Twitter) instance is https://X (Twitter).net. You can specify alternative instances in advanced settings:

Popular instances:

• https://X (Twitter).net (default)
• https://X (Twitter).poast.org
• https://X (Twitter).privacydev.net
• https://X (Twitter).1d4.us

Limitations

• Scraping speed depends on X (Twitter) instance availability
• Some X (Twitter) instances may be rate-limited or temporarily blocked
• Historical data availability depends on X (Twitter) caching
• Residential proxies require an Apify paid plan

Best Practices

1. Use Residential Proxies - For better reliability and avoiding rate limits
2. Set Reasonable Limits - Use maxItems and maxPages to control scope
3. Respect Delays - The Actor includes automatic delays; don't circumvent them
4. Monitor Runs - Check logs for any issues with X (Twitter) instances
5. Rotate Instances - If one instance fails, try another in advanced settings
6. Start Small - Test with small limits before scaling up

🔐 Privacy & Ethics

This Actor uses X (Twitter), a privacy-focused Twitter frontend. Please ensure you:

• ✅ Comply with Twitter's Terms of Service
• ✅ Respect user privacy and data protection laws (GDPR, CCPA, etc.)
• ✅ Don't scrape personal/sensitive information without proper consent
• ✅ Use scraped data responsibly and ethically
• ✅ Implement appropriate rate limiting
• ✅ Don't overload servers or abuse the service

🐛 Troubleshooting

Issue: No data returned

Solutions:

• Check if the X (Twitter) instance is working (visit X (Twitter).net in browser)
• Try a different X (Twitter) instance in advanced settings
• Enable residential proxy if not already enabled
• Verify Twitter usernames exist and are spelled correctly

Issue: Rate limiting errors

Solutions:

• Increase delays between requests (default is already 4-8 seconds)
• Enable residential proxy
• Reduce maxPages parameter
• Try a different X (Twitter) instance

Issue: Proxy errors

Solutions:

• Verify proxy configuration is correct
• Ensure you have Apify proxy credits available
• Try different proxy groups/countries
• Check proxy group availability in Apify Console

Issue: Empty media results

Solutions:

• Ensure the profile actually has media tweets
• Try without mediaOnly filter to see all tweets first
• Check if the X (Twitter) instance properly displays media

Issue: Search returns no results

Solutions:

• Verify search terms are spelled correctly
• Try broader search terms
• Remove date filters to expand search range
• Try different search type (tweets vs users)

📚 Resources

• X (Twitter) GitHub
• [X (Twitter) Instances List](https://github.com/zedeus/X (Twitter)/wiki/Instances)
• Apify Documentation
• Apify SDK for Python

🤝 Support

For issues, questions, or feature requests:

• Contact Apify support
• Check the Actor's logs for detailed error messages
• Review this documentation for troubleshooting steps

📄 License

This Actor is provided as-is for use on the Apify platform. Please ensure compliance with all applicable laws, terms of service, and ethical guidelines when using this tool.

🎯 Quick Start

1. Add Twitter usernames, list URLs, or search terms
2. Configure filters (optional)
3. Enable residential proxy for better results (optional)
4. Click "Start" - the Actor handles the rest automatically!

No mode selection needed - just provide your inputs and go! 🚀