Twitter (X.com) Tweets & Profiles Scraper

A powerful Apify Actor for scraping tweets, profiles, and search results from Twitter/X.com. This actor extracts comprehensive tweet data including text, media, engagement metrics, and user information.

Why Choose Us?

• Comprehensive Data Extraction: Get complete tweet data including text, media, links, engagement metrics, and user profiles
• Flexible Input Options: Support for profile URLs, usernames, and keyword searches
• Smart Proxy Management: Automatic fallback from no proxy → datacenter → residential proxy with retry logic
• Incremental Data Saving: Results are saved incrementally to prevent data loss
• Robust Error Handling: Graceful error handling with detailed logging
• Bulk Processing: Process multiple profiles/keywords in a single run

Key Features

• ✅ Scrape tweets from user profiles
• ✅ Search tweets by keywords
• ✅ Extract media (images, videos)
• ✅ Get engagement metrics (likes, retweets, replies, quotes)
• ✅ Include or exclude user profile information
• ✅ Filter replies option
• ✅ Automatic proxy fallback for blocked requests
• ✅ Real-time progress logging
• ✅ Structured JSON output

Input

Input Schema

{
  "startUrls": [
    { "url": "https://x.com/username" },
    { "url": "username" },
    { "url": "search keyword" }
  ],
  "maxTweets": 10,
  "withReplies": true,
  "includeUserInfo": true,
  "sortOrder": "newest",
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Input Fields

• startUrls (required): Array of Twitter/X profile URLs, usernames, or search keywords

  • Profile URLs: https://x.com/username or x.com/username
  • Usernames: username or @username
  • Keywords: Any search term (e.g., "python programming")

• maxTweets (optional, default: 10): Maximum number of tweets to scrape per profile/keyword (1-100)

• withReplies (optional, default: true): Include reply tweets in results

• includeUserInfo (optional, default: true): Include detailed user profile information with each tweet

• sortOrder (optional, default: "newest"): How to sort tweets

  • Options: "newest", "oldest", "popular"

• proxyConfiguration (optional): Proxy settings

  • By default, uses no proxy
  • If requests are blocked, automatically falls back to datacenter proxy
  • If datacenter fails, falls back to residential proxy with 3 retries
  • Once fallback occurs, uses residential proxy for all remaining requests

Output

Output Format

Each tweet is saved as a separate item in the dataset with the following structure:

{
  "id": "1519480761749016577",
  "url": "https://x.com/elonmusk/status/1519480761749016577",
  "verified": true,
  "username": "@elonmusk",
  "fullname": "Elon Musk",
  "timestamp": "Thu Apr 28 00:56:58 +0000 2022",
  "text": "Next I'm buying Coca-Cola to put the cocaine back in",
  "links": [],
  "isPinned": false,
  "isQuote": false,
  "isRetweet": false,
  "isReply": false,
  "likes": 4289224,
  "replies": 170050,
  "retweets": 594428,
  "quotes": 167104,
  "user": {
    "avatar": "https://pbs.twimg.com/profile_images/...",
    "username": "elonmusk",
    "userFullName": "Elon Musk",
    "description": "",
    "location": null,
    "website": null,
    "joinDate": "Tue Jun 02 20:12:29 +0000 2009",
    "verified": true,
    "totalLikes": 182734,
    "totalTweets": 89153,
    "totalFollowing": 1226,
    "totalFollowers": 229037864,
    "url": null,
    "userId": "44196397",
    "totalMediaCount": 4225
  },
  "media": [
    {
      "type": "photo",
      "url": "https://pbs.twimg.com/media/..."
    }
  ],
  "tweetUserId": "44196397"
}

Output Fields

• id: Tweet ID
• url: Direct link to the tweet
• verified: Whether the user is verified
• username: Twitter username with @
• fullname: User's display name
• timestamp: Tweet creation timestamp
• text: Full tweet text
• links: Array of URLs found in the tweet
• isPinned: Whether the tweet is pinned
• isQuote: Whether it's a quote tweet
• isRetweet: Whether it's a retweet
• isReply: Whether it's a reply
• likes: Number of likes
• replies: Number of replies
• retweets: Number of retweets
• quotes: Number of quote tweets
• user: User profile information (if includeUserInfo is true)
• media: Array of media items (photos, videos)
• tweetUserId: User ID of the tweet author

🚀 How to Use the Actor (via Apify Console)

1. Log in at https://console.apify.com and go to Actors
2. Find your actor (twitter-x-tweets-profiles-scraper) and click it
3. Configure inputs:
   • Add profile URLs, usernames, or keywords in startUrls
   • Set maxTweets (1-100)
   • Toggle withReplies and includeUserInfo as needed
   • Configure proxyConfiguration if needed
4. Run the actor
5. Monitor logs in real time to see progress
6. Access results in the OUTPUT tab
7. Export results to JSON or CSV

Best Use Cases

• Social Media Monitoring: Track tweets from specific users or topics
• Content Research: Collect tweets for content analysis
• Trend Analysis: Gather data on trending topics
• Influencer Research: Analyze influencer tweet patterns
• Brand Monitoring: Track mentions and engagement
• Academic Research: Collect social media data for studies

Frequently Asked Questions

Q: How many tweets can I scrape?
A: The actor supports up to 100 tweets per profile/keyword. You can process multiple profiles/keywords in a single run.

Q: What happens if I get blocked?
A: The actor automatically falls back to datacenter proxy, then residential proxy with retries. All proxy events are logged clearly.

Q: Can I scrape private accounts?
A: No, the actor only scrapes publicly available tweets and profiles.

Q: How long does scraping take?
A: Depends on the number of profiles/keywords and maxTweets setting. Typically 1-5 minutes per profile.

Q: Is the data real-time?
A: Yes, the actor scrapes current data from X.com at the time of execution.

Q: Can I filter out replies?
A: Yes, set withReplies: false in the input to exclude reply tweets.

Support and Feedback

For issues, questions, or feedback, please contact support or open an issue in the actor repository.

Cautions

• Data is collected only from publicly available sources
• No data is taken from private accounts or password-protected content
• The end user is responsible for ensuring legal compliance (spam laws, privacy, data protection, etc.)
• Respect Twitter/X's Terms of Service and rate limits
• Use responsibly and ethically