🌸 Tweets / X - Scraper

Extract anything from X (Twitter) with speed and precision. This smart scraper auto-detects what to collect—tweets, profiles, users, lists, or media—and delivers clean, structured data instantly. Just enter usernames, URLs, or keywords and let automation do the rest.

Extract anything from X (Twitter) with speed and precision. This smart scraper auto-detects what to collect—tweets, profiles, users, lists, or media—and delivers clean, structured data instantly. Just enter usernames, URLs, or keywords and let automation do the rest.

---

🚀 Features

Smart Automatic Routing

The Actor automatically detects what you want to scrape based on your inputs:

• Twitter Usernames → Automatically scrapes profiles (smart mode detection)
• Twitter List URLs → Automatically scrapes list content
• Tweet URLs → Automatically scrapes specific tweets and their entire threads
• Search Terms → Automatically searches tweets or users

Intelligent Profile Scraping

When scraping profiles, the Actor automatically selects the best mode:

• ✅ Media Only Mode — Check Media Only to extract only tweets with images/videos
• ✅ With Replies Mode — Enable Include Replies to get all tweets including replies
• ✅ Normal Mode — Standard tweet scraping (default)
• ✅ Profile Search — Add search terms to find specific content within profiles

Advanced Features

• 🎯 Advanced Filtering — Filter by date, location, media type, engagement metrics
• 👤 Profile Information — Extract complete profile details (bio, stats, avatar, banner)
• 📷 Media Extraction — Get direct URLs for images and videos with HLS preview links
• 📊 Engagement Stats — Collect likes, retweets, comments, and quotes
• 🔄 Rate Limiting Protection — Built-in retry logic and exponential backoff
• 📄 Pagination — Automatic handling of multiple pages
• 📋 Structured Output — Clean JSON output to Apify dataset

---

📋 Input Configuration

Main Input Fields

1. Twitter Usernames

Add one or more Twitter usernames to scrape profiles. The Actor will automatically:

• Choose between normal, with-replies, or media-only mode based on your settings No mode selection needed - just provide your inputs and go! 🚀by minimum likes (minLikes), minimum retweets (minRetweets), minimum replies (minReplies), verified authors only (onlyVerified), quotes only (onlyQuotes), or tweets with external links (includeLinks).
• Rich Media & Content: Extracts full-resolution images, original MP4 videos, animated GIFs, video poster thumbnails, and Twitter card previews.
• Anti-Blocking Built-In: Integrated support for Apify Residential proxies and intelligent session recycling.

<<<<<<< HEAD Examples:

elonmusk
apify
@naval (@ symbol is optional)

=======

complete-suits

📋 Input Parameters

<<<<<<< HEAD Supported formats:

https://x.com/i/lists/123456
https://twitter.com/i/lists/123456

3. Tweet URLs

Add specific Twitter/X tweet URLs to extract the tweet, its full thread, and replies.

Examples:

https://x.com/apify/status/123456789
https://twitter.com/user/status/123456789

4. Search Terms

Add search queries to find tweets or users across Twitter.

Examples:

web scraping
AI technology
from:apify
#python

---

Profile Options

• Include Replies — Include reply tweets (automatically switches to "with replies" mode)
• Include Retweets — Include retweets in results
• Media Only — Only scrape tweets containing images or videos (uses dedicated media scraper)

Search Options

When using Search Terms, choose what to search:

• Search Tweets Globally — Find tweets across all Twitter
• Search Users — Find Twitter accounts
• Search Within Profiles — Search for specific content in provided profiles (requires Twitter Usernames)

Content Filters

Filter tweets by content characteristics:

• Only Images — Filter tweets containing images
• Only Videos — Filter tweets containing videos
• Only Verified Users — Only include verified accounts
• Only Quote Tweets — Only include quote tweets
• Include Native Retweets — Include native retweets
• Only Tweets with Links — Filter tweets with external links
• Safe Search — Enable to filter sensitive content

Date & Location Filters

• Start Date (From) — Filter tweets from this date onwards (YYYY-MM-DD format)
• End Date (Until) — Filter tweets up to this date (YYYY-MM-DD format)
• Location Filter — Filter by location (e.g., "Los Angeles", "New York")

Engagement Filters

Set minimum engagement thresholds:

• Minimum Likes — Minimum number of likes required
• Minimum Replies — Minimum number of replies required
• Minimum Retweets — Minimum number of retweets required

Limits & Pagination

• Max Items to Scrape — Maximum items (tweets/users) to collect (0 = unlimited for paid users; max 10 for free users)
• Max Retries — Number of retry attempts for failed requests

Advanced Options

• Tweet Language — Filter by language code (e.g., en, es, fr)
• Scrape Profile Info — Extract detailed profile information (bio, stats, avatar, etc.)

---

🔧 Usage Examples

Example 1: Simple Profile Scraping

=======

Parameter	Type	Default	Description
twitterHandles	Array	["rish_w3b"]	Twitter handles to scrape (with or without @).
listUrls	Array	[]	Twitter list URLs (e.g. https://x.com/i/lists/123456).
searchTerms	Array	[]	Search terms, keywords, hashtags, or phrases.
searchType	String	"tweets"	tweets (Global Tweet Search), users (User Search), or profile_tweets (Search Within Profiles).
includeReplies	Boolean	false	Include replies when scraping profiles.
includeRetweets	Boolean	true	Include retweets in results.
mediaOnly	Boolean	false	Only scrape tweets containing images or videos.
maxItems	Integer	100	Maximum items to scrape across all targets (0 = unlimited for paid users; max 10 for free users).
startDate	String	""	Start date filter (YYYY-MM-DD).
endDate	String	""	End date filter (YYYY-MM-DD).
location	String	""	Location filter (e.g. 'New York', 'London').
onlyImages	Boolean	false	Filter tweets that contain images.
onlyVideos	Boolean	false	Filter tweets that contain videos.
onlyVerified	Boolean	false	Only tweets from verified accounts.
onlyQuotes	Boolean	false	Only quote tweets.
includeNativeRetweets	Boolean	true	Include native retweets in search.
includeLinks	Boolean	false	Only tweets containing external URLs.
latestTweets	Boolean	false	Only scrape tweets from the last 7 days.
safeSearch	Boolean	false	Enable safe search to filter sensitive content.
minLikes	Integer	0	Minimum likes threshold.
minReplies	Integer	0	Minimum replies threshold.
minRetweets	Integer	0	Minimum retweets threshold.
tweetLanguage	String	""	Language code (e.g. 'en', 'es', 'fr').
scrapeProfileInfo	Boolean	true	Extract full profile metadata header when scraping handles.
useResidentialProxy	Boolean	false	Use Apify Residential Proxy for maximum reliability.
maxRetries	Integer	5	Retry attempts per failed page request.

---

💡 Example Configurations

complete-suits

1. Scrape Profile Timelines with Media

{
  "twitterHandles": ["elonmusk", "OpenAI"],
  "scrapeProfileInfo": true,
  "mediaOnly": false,
  "maxItems": 100
}

<<<<<<< HEAD What happens: Scrapes the last 100 tweets from each profile using normal mode.

Example 2: Profile with Replies

=======

2. Scrape a Twitter List

complete-suits

{
  "listUrls": [
    "https://x.com/i/lists/123456789"
  ],
  "maxItems": 200
}

<<<<<<< HEAD What happens: Automatically switches to "with replies" mode to get all tweets including replies.

Example 3: Media Only Scraping

{
  "twitterHandles": ["apify", "elonmusk"],
  "mediaOnly": true
}

What happens: Uses the dedicated media scraper to extract only tweets with images/videos.

Example 4: Global Tweet Search with Filters

{
  "searchTerms": ["web scraping", "#ai"],
=======

### 3. High-Engagement Topic Search
```json
{
  "searchTerms": ["artificial intelligence", "#LLM"],
>>>>>>> complete-suits
  "searchType": "tweets",
  "minLikes": 50,
  "tweetLanguage": "en",
  "startDate": "2024-01-01",
  "maxItems": 500
}

<<<<<<< HEAD What happens: Searches for tweets globally with engagement and verification filters.

Example 5: Profile Search (Find Specific Content)

{
  "twitterHandles": ["apify"],
  "searchTerms": ["scraping", "automation"],
  "searchType": "profile_tweets",
  "startDate": "2024-01-01"
}

What happens: Searches for specific keywords within the provided profile's tweets.

Example 6: Scrape Twitter List

{
  "listUrls": ["https://x.com/i/lists/78783491"],
  "maxItems": 200
}

What happens: Scrapes up to 200 tweets from the specified list.

Example 7: User Search

{
  "searchTerms": ["data scientist", "ML engineer"],
=======

### 4. Search Users by Topic
```json
{
  "searchTerms": ["crypto founder", "AI researcher"],
>>>>>>> complete-suits
  "searchType": "users",
  "maxItems": 50
}

<<<<<<< HEAD What happens: Finds Twitter users matching the search terms.

---

complete-suits

📊 Output Dataset Format

Tweet Item

{
<<<<<<< HEAD
  "twitterHandles": ["apify"],
  "listUrls": ["https://x.com/i/lists/123456"],
  "searchTerms": ["web scraping"],
  "searchType": "tweets",
  "maxItems": 100
}

What happens: Scrapes all three: profile tweets, list tweets, and search results.

---

📊 Output Format

Data is saved to the Apify dataset with structured JSON objects:

Profile Output

{
  "type": "profile",
  "username": "apify",
  "fullname": "Apify",
  "bio": "Web scraping and automation platform...",
  "stats": {
    "tweets": "1,234",
    "followers": "12.5K",
    "following": "567"
  },
  "avatar": "https://pbs.twimg.com/...",
  "banner": "https://pbs.twimg.com/...",
  "website": "https://apify.com",
  "location": "San Francisco, CA",
  "joined": "Joined January 2024",
  "scrape_date": "2026-08-25T12:00:00.000Z"
}

Tweet Output

{
  "type": "tweet",
  "tweet_url": "https://x.com/apify/status/123456",
  "tweet_id": "123456",
  "profile_username": "apify",
  "author": {
    "fullname": "Apify",
    "username": "@apify",
    "avatar": "https://pbs.twimg.com/...",
    "verified": true,
    "verified_type": "blue"
  },
  "text": "Tweet content here...",
  "date": "Oct 24, 2024 · 12:00 PM UTC",
  "date_short": "Oct 24",
  "hashtags": ["#webscraping", "#automation"],
  "mentions": ["@user1", "@user2"],
  "urls": ["https://example.com"],
  "stats": {
    "comments": 12,
    "retweets": 34,
    "quotes": 5,
    "likes": 156
  },
  "media": [
    {
      "type": "video",
      "url": "https://video.twimg.com/...",
      "hls_preview": "https://hlsjs.video-dev.org/demo/?src=...",
      "thumbnail": "https://pbs.twimg.com/...",
      "requires_hls": false
    },
    {
      "type": "image",
      "url": "https://pbs.twimg.com/...",
      "thumbnail": "https://pbs.twimg.com/..."
    }
  ],
  "is_retweet": false,
  "scrape_date": "2026-08-25T12:00:00.000Z"
}

Media Tweet Output

{
  "type": "media_tweet",
  "profile_username": "apify",
  "tweet_id": "123456",
  "media": [
    {
      "type": "image",
      "url": "https://pbs.twimg.com/...",
      "thumbnail": "https://pbs.twimg.com/..."
    }
  ],
  "stats": {
    "comments": 5,
    "retweets": 10,
    "quotes": 2,
    "likes": 45
  },
  "scrape_date": "2026-08-25T12:00:00.000Z"
}

User Search Output

{
  "type": "user",
  "search_query": "data scientist",
  "username": "@john_doe",
  "fullname": "John Doe",
  "display_name": "John Doe",
  "handle": "@john_doe",
  "bio": "Data Scientist | ML Engineer...",
  "avatar": "https://pbs.twimg.com/...",
  "verified": true,
  "verified_type": "blue",
  "protected": false,
  "hashtags": ["#DataScience", "#ML"],
  "mentions": [],
  "urls": ["https://johndoe.com"],
  "profile_url": "https://x.com/john_doe",
  "scrape_date": "2026-08-25T12:00:00.000Z"
=======
  "id": "1892837465928172635",
  "type": "tweet",
  "url": "https://x.com/elonmusk/status/1892837465928172635",
  "text": "Exciting advances in AI systems coming this year.",
  "timestamp": "2026-08-21T16:20:00Z",
  "username": "elonmusk",
  "fullname": "Elon Musk",
  "likes": 42500,
  "retweets": 8300,
  "replies": 3200,
  "quotes": 940,
  "has_media": true,
  "images": ["https://pbs.twimg.com/media/xyz123.jpg"],
  "videos": [],
  "urls": ["https://x.ai"],
  "hashtags": ["AI"],
  "mentions": [],
  "author": {
    "username": "elonmusk",
    "fullname": "Elon Musk",
    "verified": true,
    "verified_type": "blue",
    "profile_picture": "https://pbs.twimg.com/profile_images/.../photo.jpg"
  }
}

Profile Metadata Item

{
  "id": "elonmusk",
  "type": "profile",
  "username": "elonmusk",
  "fullname": "Elon Musk",
  "bio": "...",
  "location": "Texas",
  "website": "https://x.com",
  "joined": "June 2009",
  "followers": 180000000,
  "following": 500,
  "tweets_count": 45000,
  "banner": "https://pbs.twimg.com/profile_banners/.../banner.jpg",
  "profile_picture": "https://pbs.twimg.com/profile_images/.../photo.jpg",
  "verified": true,
  "verified_type": "blue"
>>>>>>> complete-suits
}

---

<<<<<<< HEAD

🛠️ Technical Details

Architecture

The Actor uses:

• Smart Routing Logic — Automatically detects scraping mode based on inputs
• Headless Chromium Execution — Playwright-powered browser automation to bypass anti-bot challenges
• Apify SDK for Python — Full integration with Apify platform, Key-Value stores, and Datasets

Automatic Mode Detection

The Actor intelligently routes requests:

IF twitterHandles PROVIDED:
  IF mediaOnly = true → profile_media
  ELSE IF includeReplies OR includeRetweets → profile_with_reply
  ELSE → profile_main
  
  IF searchTerms PROVIDED AND searchType = "profile_tweets" → profile_search

IF listUrls PROVIDED → list_extractor

IF searchTerms PROVIDED (without twitterHandles):
  IF searchType = "tweets" → search_post
  ELSE IF searchType = "users" → search_user

Rate Limiting & Proxy Support

• Automatic retry logic with exponential backoff
• Direct fast connection by default with automatic fallback to US/CA residential proxies on connection issues

---

📝 Notes & Best Practices

1. Set Reasonable Limits — Use maxItems to control execution scope.
2. Monitor Runs — Check Apify run logs for real-time progress.
3. Start Small — Test with small sample limits before scaling up large scraping tasks.

---

🔐 Privacy & Ethics

Please ensure you:

• ✅ Comply with Twitter's Terms of Service
• ✅ Respect user privacy and data protection laws (GDPR, CCPA, etc.)
• ✅ Use scraped data responsibly and ethically

---

🐛 Troubleshooting

Issue: No data returned

Solutions:

• Verify Twitter usernames exist and are spelled correctly
• Try broadening filters or removing date limits

Issue: Empty media results

Solutions:

• Ensure the profile actually contains tweets with media attachments
• Try without mediaOnly filter to see all tweets first

Issue: Search returns no results

Solutions:

• Verify search terms and hashtags are spelled correctly
• Remove strict date or engagement filters to expand search range

---

📚 Resources

• Apify Documentation
• Apify SDK for Python

---

🤝 Support

For issues, questions, or feature requests:

• Contact support via the Apify Console
• Check the Actor's logs for detailed run messages

---

📄 License

This Actor is provided for use on the Apify platform. Please ensure compliance with all applicable laws and ethical guidelines when using this tool.

---

🎯 Quick Start

1. Add Twitter usernames, list URLs, or search terms
2. Configure optional filters
3. Click "Start" — the Actor handles the rest automatically! 🚀 =======

⚖️ License

This project is licensed under the AGPL-3.0 License.

complete-suits