🛠️ X / Twitter Scraper (Ultimate)

Professional-grade Twitter/X data extraction tool that delivers structured JSON data instantly. Perfect for social media monitoring, lead generation, content research, and competitive analysis.

---

What You Can Extract

📱 Profile Data

• Complete user profiles with bio, location, website, avatar, banner
• Follower/following counts and tweet statistics
• Join dates and verification status
• Profile metadata for lead generation

🐦 Tweet Content

• Full tweet text with hashtags and mentions
• Media URLs (images, videos with HLS preview links)
• Engagement metrics (likes, retweets, comments, quotes)
• Timestamps and tweet URLs
• Author information with verification details

🔍 Search Capabilities

• Global Tweet Search: Find tweets across all of Twitter/X
• User Search: Discover accounts matching your criteria
• Profile Search: Search within specific user profiles

📋 List Scraping

• Extract all tweets from Twitter Lists
• Track curated content collections
• Monitor list updates automatically

---

Key Features

✨ Intelligent Auto-Detection - Automatically determines the best scraping mode based on your inputs
🎯 Advanced Filtering - Filter by date range, location, media type, engagement, language, and more
🔒 Built-in Residential Proxy - Reliable data extraction with automatic proxy management
📊 Multiple Data Views - Pre-configured views for profiles, tweets, media, and engagement analysis
⚡ Streaming Results - Real-time data delivery as scraping progresses
🔄 Smart Retry Logic - Automatic retries with exponential backoff for failed requests
📈 Engagement Analytics - Filter by minimum likes, retweets, or replies

---

Input Parameters

Core Inputs

Twitter Usernames (twitterHandles)

• Enter one or more usernames (with or without @)
• Example: elonmusk, apify, @naval
• Automatically scrapes profile tweets

List URLs (listUrls)

• Twitter/X list URLs in format: https://x.com/i/lists/123456
• Scrapes all tweets from the specified lists

Search Terms (searchTerms)

• Search queries to find tweets or users
• Examples: "web scraping", "#AI", "from:apify"
• Works with global search or profile-specific search

Search Configuration

Search Type (searchType)

• tweets - Search tweets globally across Twitter/X
• users - Find Twitter accounts
• profile_tweets - Search within specific profiles (requires usernames)

Profile Options

• Include Replies - Get reply tweets along with regular tweets
• Include Retweets - Include retweeted content in results
• Media Only - Extract only tweets containing images or videos
• Scrape Profile Info - Extract complete profile details (default: enabled)

Content Filters

Media Filters:

• Only Images - Tweets with images only
• Only Videos - Tweets with videos only
• Only Tweets with Links - Filter tweets containing external links

Account Filters:

• Only Verified Users - Limit to verified accounts
• Only Quote Tweets - Extract quote tweets exclusively
• Include Native Retweets - Include native retweet format (default: enabled)
• Safe Search - Filter sensitive content

Time & Location Filters

• Start Date - Filter from date (YYYY-MM-DD format)
• End Date - Filter until date (YYYY-MM-DD format)
• Location - Filter by geographic location (e.g., "Los Angeles", "New York")

Engagement Filters

Set minimum thresholds:

• Minimum Likes - Filter tweets by minimum like count
• Minimum Replies - Filter tweets by minimum reply count
• Minimum Retweets - Filter tweets by minimum retweet count

Language & Limits

• Tweet Language - Filter by language code (e.g., en, es, fr, de)
• Max Items - Maximum items to scrape (0 = unlimited, default: 100)
• Max Pages - Maximum pages per handle/query (default: 100, max: 100)
• Max Retries - Retry attempts for failed requests (1-10, default: 5)

---

Output Data Structure

All scraped data is saved to the Apify dataset in structured JSON format with multiple pre-configured views:

Available Views

1. All Results - Complete dataset with all fields
2. Profiles Only - Profile information view
3. Tweets Only - Tweet/post data view
4. Tweets with Media - Media-rich tweets
5. Top Engagement - Tweets sorted by engagement metrics

Data Fields

Profile Objects Include:

• username, fullname, bio, location, website
• joined date, avatar, banner URLs
• stats object with tweets, followers, following counts
• scrape_date timestamp

Tweet Objects Include:

• tweet_url, tweet_id, text, date
• author object with profile details
• hashtags, mentions, urls arrays
• stats object with engagement metrics
• media array with image/video URLs and metadata
• is_retweet boolean flag
• scrape_date timestamp

Search Results Include:

• search_query field for search context
• type field indicating data type (profile, tweet, user, etc.)

---

Usage Examples

Example 1: Extract Profile Tweets

{
  "twitterHandles": ["elonmusk", "apify"],
  "maxItems": 100
}

Scrapes the latest 100 tweets from each profile.

Example 2: Media-Only Scraping

{
  "twitterHandles": ["apify"],
  "mediaOnly": true,
  "maxItems": 50
}

Extracts only tweets containing images or videos.

Example 3: Search with Filters

{
  "searchTerms": ["LLM", "AI"],
  "searchType": "tweets",
  "startDate": "2024-01-01",
  "endDate": "2024-12-31",
  "onlyVerified": true,
  "minLikes": 10,
  "tweetLanguage": "en",
  "maxItems": 200
}

Searches for verified English tweets about LLM/AI from 2024 with at least 10 likes.

Example 4: Profile Search

{
  "twitterHandles": ["apify"],
  "searchTerms": ["scraping", "automation"],
  "searchType": "profile_tweets",
  "maxItems": 100
}

Searches for specific keywords within a profile's tweets.

Example 5: User Discovery

{
  "searchTerms": ["data scientist", "ML engineer"],
  "searchType": "users",
  "maxItems": 50
}

Finds Twitter accounts matching the search terms.

Example 6: List Scraping

{
  "listUrls": ["https://x.com/i/lists/78783491"],
  "maxItems": 200
}

Scrapes tweets from a Twitter List.

Example 7: Engagement-Based Filtering

{
  "searchTerms": ["web scraping"],
  "searchType": "tweets",
  "minLikes": 50,
  "minRetweets": 10,
  "onlyImages": true,
  "maxItems": 100
}

Finds high-engagement image tweets about web scraping.

Example 8: Complete Profile with Replies

{
  "twitterHandles": ["elonmusk"],
  "includeReplies": true,
  "includeRetweets": true,
  "scrapeProfileInfo": true,
  "maxItems": 500
}

Comprehensive profile extraction including replies and retweets.

---

Output Views Explained

The dataset includes 5 pre-configured views for easy data analysis:

📊 All Results View

Displays complete dataset with all available fields. Perfect for comprehensive data export.

👤 Profiles Only View

Filtered view showing only profile information. Ideal for lead generation and user research.

🐦 Tweets Only View

Tweet-focused view sorted by date (newest first). Best for content analysis and trend monitoring.

🎬 Tweets with Media View

Media-rich tweets only. Perfect for visual content analysis and media asset collection.

📈 Top Engagement View

Tweets sorted by engagement metrics (likes + retweets). Great for identifying viral content.

---

Technical Specifications

Supported Operations

• Profile timeline scraping (normal, with replies, media-only modes)
• Twitter List extraction
• Global tweet search
• User account search
• Profile-specific search
• Advanced filtering and pagination

Data Extraction

• Automatic pagination handling
• Real-time streaming results
• Structured JSON output
• Multiple data views for different use cases

Reliability Features

• Built-in residential proxy support (automatic)
• Configurable retry logic (1-10 attempts)
• Error handling and graceful degradation
• Rate limiting protection

---

Use Cases

🎯 Lead Generation

Extract verified users, filter by engagement, and collect profile data for outreach campaigns.

📊 Social Media Monitoring

Track mentions, hashtags, and trending topics with date and location filters.

🔍 Competitive Analysis

Monitor competitor profiles, analyze engagement patterns, and track content strategies.

📈 Content Research

Find high-performing tweets, analyze media content, and identify trending topics.

🎬 Media Collection

Extract images and videos from profiles or search results with direct URLs.

👥 User Discovery

Search for users by keywords, industry, or interests for targeted outreach.

---

Best Practices

1. Start Small - Test with maxItems: 50 before scaling up
2. Use Filters - Apply engagement filters to get high-quality results
3. Date Ranges - Use date filters to focus on specific time periods
4. Residential Proxy - Enabled by default for reliable extraction
5. Monitor Limits - Set reasonable maxPages and maxItems to control costs
6. Language Filtering - Use tweetLanguage to target specific regions
7. Profile Info - Enable scrapeProfileInfo for complete user data

---

Data Privacy & Compliance

This tool is designed for public data extraction. Users are responsible for:

• Compliance with Twitter/X Terms of Service
• Adherence to data protection regulations (GDPR, CCPA, etc.)
• Ethical use of scraped data
• Respect for user privacy and consent requirements

---

Support & Resources

For technical support, feature requests, or questions:

• Check the Actor logs for detailed error messages
• Review input parameters in the Actor interface
• Consult Apify documentation for platform-specific questions

---

Version

Current Version: 1.0
Categories: Social Media, Lead Generation, Automation
Platform: Apify

---

Ready to extract Twitter/X data? Configure your inputs, set your filters, and start scraping. The Actor handles everything automatically! 🚀