Twitter / X Scraper — Profiles, Tweets & Search

Scrape Twitter/X profiles, tweets, and search results at scale. Extract engagement metrics, media URLs, profile data, and full tweet text — all without API keys or Twitter developer accounts.

Features

• Profile scraping: Display name, bio, follower/following counts, tweet count, verified status, join date, location, website, avatar
• Tweet extraction: Full text, likes, retweets, replies, quotes, views, date, media (images/videos), hashtags, mentions, URLs
• Search: Find tweets by keyword, hashtag, or Twitter search operators
• Multi-strategy: Uses syndication API, Nitter mirrors, and direct page parsing with automatic fallbacks
• Rate limit handling: Automatic exponential backoff, request delays, and retry logic
• User-Agent rotation: 10+ realistic browser fingerprints to avoid detection
• Deduplication: Automatically removes duplicate tweets across strategies
• Language filter: Filter tweets by ISO 639-1 language code
• Proxy support: Full compatibility with Apify Proxy for high-volume scraping
• Pay Per Event: Only pay for data you actually receive

Use Cases

Social Listening

Monitor what people are saying about your brand, product, or industry in real-time. Track sentiment and trending topics across Twitter.

Brand Monitoring

Set up regular scrapes to track mentions of your brand name, product names, or key executives. Get alerts on viral content before it peaks.

Competitor Analysis

Analyze competitor Twitter profiles — their posting frequency, engagement rates, content strategy, and audience growth. Identify what content resonates with their audience.

Influencer Research

Find and evaluate potential influencer partners. Analyze their follower counts, engagement rates, content themes, and audience demographics.

Market Research

Search for industry-specific conversations, pain points, and feature requests. Identify emerging trends before they go mainstream.

Lead Generation

Find potential customers discussing problems your product solves. Monitor buying signals and intent keywords.

Academic Research

Collect structured tweet datasets for NLP, sentiment analysis, network analysis, and social science research.

Content Strategy

Analyze top-performing tweets in your niche. Identify optimal posting times, content formats, hashtags, and engagement patterns.

Input

Field	Type	Default	Description
handles	string[]	[]	Twitter usernames to scrape (without @ prefix)
searchTerms	string[]	[]	Keywords or phrases to search for
maxTweets	integer	20	Max tweets per handle or search term (1-500)
includeReplies	boolean	false	Include reply tweets in results
tweetLanguage	string	-	Filter by language (ISO 639-1: 'en', 'es', 'ja', etc.)
scrapeProfile	boolean	true	Extract full profile info for each handle
includeMedia	boolean	true	Include image/video URLs from tweets
proxy	object	-	Proxy config (recommended for Twitter)

Example: Scrape Profiles & Tweets

{
  "handles": ["elonmusk", "OpenAI", "AnthropicAI"],
  "maxTweets": 50,
  "includeReplies": false,
  "scrapeProfile": true
}

Example: Search by Keyword

{
  "searchTerms": ["AI agents", "#buildinpublic", "MCP server"],
  "maxTweets": 100,
  "tweetLanguage": "en"
}

Example: Competitor Analysis

{
  "handles": ["stripe", "square", "paypal"],
  "searchTerms": ["payment API review", "stripe vs square"],
  "maxTweets": 30,
  "scrapeProfile": true,
  "includeMedia": false
}

Example: Influencer Research

{
  "handles": ["levelsio", "marc_louvion", "dannypostmaa"],
  "maxTweets": 100,
  "scrapeProfile": true,
  "includeReplies": true
}

Output

Profile Object

{
  "type": "profile",
  "username": "elonmusk",
  "displayName": "Elon Musk",
  "bio": "Mars & Cars, Chips & Dips",
  "followerCount": 178500000,
  "followingCount": 782,
  "tweetCount": 45200,
  "likesCount": 52300,
  "verified": true,
  "avatar": "https://pbs.twimg.com/profile_images/.../photo.jpg",
  "joinDate": "Jun 2009",
  "location": "Earth",
  "website": "https://x.com",
  "profileUrl": "https://x.com/elonmusk",
  "scrapedAt": "2026-03-01T12:00:00.000Z",
  "scrapeStrategy": "nitter"
}

Tweet Object

{
  "type": "tweet",
  "tweetId": "1895123456789012345",
  "username": "elonmusk",
  "text": "The future of AI is incredibly exciting. We're building something special.",
  "likes": 245000,
  "retweets": 32100,
  "replies": 18500,
  "quotes": 4200,
  "views": 52000000,
  "date": "2026-02-28T18:30:00.000Z",
  "tweetUrl": "https://x.com/elonmusk/status/1895123456789012345",
  "media": [
    {
      "type": "photo",
      "url": "https://pbs.twimg.com/media/example.jpg"
    }
  ],
  "isRetweet": false,
  "isReply": false,
  "language": "en",
  "source": "Twitter for iPhone",
  "hashtags": ["AI"],
  "mentions": [],
  "urls": [],
  "scrapedAt": "2026-03-01T12:00:00.000Z",
  "scrapeStrategy": "syndication"
}

How It Works

This actor uses a multi-strategy approach with automatic fallbacks:

1. Twitter Syndication API — Fetches embedded timeline data from Twitter's public syndication endpoint. Fastest and most reliable for tweet content.

2. Nitter Instances — Public Twitter mirrors that serve Twitter data without JavaScript. Probes multiple Nitter instances and uses the first working one. Best for profile data and full tweet feeds.

3. Direct Page Crawling — CheerioCrawler-based fallback that attempts to parse Twitter/X pages directly, extracting embedded JSON and HTML content.

The actor automatically tries each strategy in order and deduplicates results across strategies to give you the most complete data possible.

Rate Limiting & Best Practices

• The actor automatically handles rate limiting with exponential backoff
• Requests are spaced 1.5-3.5 seconds apart to avoid detection
• 10 different User-Agent strings are rotated randomly
• Using Apify Proxy is strongly recommended for production scrapes
• For large-scale scraping (100+ handles), use residential proxies
• Start with small batches to test your configuration

Pay Per Event Pricing

Event	Price
Profile scraped	$0.005
Tweet scraped	$0.002

Examples:

• 10 profiles + 200 tweets = $0.05 + $0.40 = $0.45
• 50 profiles + 1000 tweets = $0.25 + $2.00 = $2.25
• Search (500 tweets) = $1.00

Limitations

• Twitter actively blocks scrapers — some requests may fail even with proxies
• Nitter instances can go offline at any time; the actor probes multiple instances
• Engagement metrics (likes, retweets) may not be available from all strategies
• Private/protected accounts cannot be scraped
• Twitter may serve different content based on region/IP
• Very old tweets may not be accessible through syndication or Nitter
• Search results are limited to what Twitter/Nitter return on the first page

Tips for Best Results

1. Use proxies — Twitter rate-limits aggressively by IP. Apify residential proxies work best.
2. Start small — Test with 1-2 handles and 10-20 tweets before scaling up.
3. Schedule regular runs — Set up recurring scrapes for ongoing monitoring.
4. Combine handles + search — Scrape competitor profiles AND search for brand mentions in one run.
5. Filter by language — Use tweetLanguage to focus on your target market.

Changelog

1.0.0 (2026-03-01)

• Initial release
• Multi-strategy scraping: syndication API, Nitter, direct crawling
• Profile extraction with full metadata
• Tweet extraction with engagement metrics and media
• Search functionality
• 10+ User-Agent rotation
• Automatic rate limiting and exponential backoff
• Deduplication across strategies
• Language filtering
• Pay Per Event billing

Related Actors

• Twitter Followers Scraper — Extract follower data
• Twitter Hashtag Scraper — Track hashtag trends
• Instagram Scraper — Cross-platform social data
• TikTok Scraper — TikTok posts and profiles
• Reddit Scraper — Reddit posts and comments