🚀 Complete X (Twitter) Scraper

⚡ Fast • 💰 Cost-Effective • 🎯 Smart • 🔥 Production Ready

Extract tweets, users, profiles, lists, and media from X (Twitter) with high speed, low cost, and zero manual configuration.

Built for developers, startups, AI pipelines, and data-driven products.

---

🌟 Why This Scraper?

Most scrapers are:

• ❌ Slow
• ❌ Expensive
• ❌ Hard to configure

This one is:

• ⚡ Blazing Fast – Optimized scraping engine with smart routing
• 💰 Cost Efficient – Minimal requests + optimized pagination
• 🧠 Smart Detection – No need to select modes manually
• 🔄 Fully Automated – Handles retries, pagination & errors
• 📦 Structured Output – Clean JSON ready for pipelines

---

🧠 What Makes It Unique?

🔥 Smart Auto-Detection Engine

Just provide input — the scraper decides everything:

Input	Automatically Does
Username	Scrapes profile + tweets
List URL	Scrapes list tweets
Keywords	Searches tweets or users

👉 No configuration needed. Just run.

---

🚀 Features

👤 Profile Scraping

• Full profile data (bio, followers, location, etc.)
• Tweets extraction
• Replies support
• Retweets support

---

📷 Media Scraping

• Extract images & videos
• Direct media URLs
• HLS preview support for videos

---

🔍 Advanced Search

• Global tweet search
• User discovery
• Profile-based search

---

🎯 Advanced Filtering

• 📅 Date range (from / until)
• 📊 Engagement (likes, replies, retweets)
• 🌍 Location-based filtering
• ✔️ Verified users only
• 📷 Media type filtering

---

⚙️ Built-in Reliability

• 🔄 Auto retry with exponential backoff
• 🛡️ Rate limit protection
• 🌐 Proxy rotation support
• 📄 Auto pagination

---

💡 Use Cases

This scraper is perfect for:

📊 Data Collection

• Social media analytics
• Trend tracking
• Market research

🤖 AI / ML Pipelines

• Training datasets
• Sentiment analysis
• NLP pipelines

🚀 Startups & SaaS

• Lead generation
• Influencer discovery
• Competitor analysis

📈 Growth & Marketing

• Hashtag tracking
• Engagement analysis
• Audience research

---

📋 Input Configuration

1️⃣ Scrape Profiles

{
  "twitterHandles": ["elonmusk", "nasa"],
  "maxItems": 100
}

---

2️⃣ Search Tweets

{
  "searchTerms": ["AI", "#startup"],
  "searchType": "tweets"
}

---

3️⃣ Search Users

{
  "searchTerms": ["data scientist"],
  "searchType": "users"
}

---

4️⃣ Scrape Lists

{
  "listUrls": ["https://x.com/i/lists/123456"]
}

---

5️⃣ Media Only Mode

{
  "twitterHandles": ["nasa"],
  "mediaOnly": true
}

---

⚙️ Configuration Options

Profile Options

Option	Description
includeReplies	Include replies
includeRetweets	Include retweets
mediaOnly	Only media tweets

---

Filters

Filter	Example
Date	"startDate": "2024-01-01"
Likes	"minLikes": 50
Language	"tweetLanguage": "en"
Verified	"onlyVerified": true

---

📊 Output Format

🧾 Tweet

{
  "type": "tweet",
  "tweet_id": "123456",
  "text": "AI is the future",
  "likes": 120,
  "retweets": 30,
  "comments": 10
}

---

👤 Profile

{
  "type": "profile",
  "username": "nasa",
  "followers": "10M",
  "bio": "Exploring space"
}

---

🔍 User

{
  "type": "user",
  "username": "@john_doe",
  "verified": true
}

---

⚡ Performance & Cost Advantage

Feature	This Scraper	Others
Speed	⚡ Fast	Slow
Cost	💰 Low	Expensive
Setup	🟢 Zero Config	Complex
Reliability	🛡️ High	Medium
Automation	🤖 Full	Partial

---

🧠 How It Works

Input → Smart Detection → Correct Scraper → Structured Output

Internal Flow

Usernames → Profile Scraper
Lists → List Extractor
Search → Tweet/User Search Engine

---

⚡ Best Practices

• Start with small maxItems
• Avoid aggressive scraping
• Use filters to reduce cost
• Monitor logs
• Rotate instances if needed

---

🐛 Troubleshooting

❌ No Data

• Check username spelling
• Remove strict filters
• Try different query

---

⚠️ Rate Limit

• Reduce pages
• Increase delay
• Use proxies

---

📷 No Media Found

• Disable mediaOnly
• Check if profile has media

---

🔐 Ethics & Legal

• Follow platform policies
• Respect privacy laws (GDPR, etc.)
• Avoid scraping sensitive/private data
• Use responsibly

---

🎯 Quick Start

1. Add input (username / search / list)
2. Set filters (optional)
3. Run scraper 🚀

---

🧩 Tech Stack

• Python Scraper Engine
• Apify SDK
• Proxy Rotation System
• Smart Routing Logic

---

💬 Support

• Check logs
• Review documentation
• Raise issues

---

📄 License

MIT License – Free to use with responsibility.

---

🔥 Final Note

This is a complete, scalable, and cost-efficient X scraper built for:

• Developers
• Startups
• AI Builders
• Growth Hackers

---