X (Twitter) User profile and Timeline scraper

This Apify Actor scrapes user timelines from X (Twitter) to extract tweets, user information, and engagement metrics. It works in guest mode without requiring authentication and can retrieve tweets from any public Twitter account.

Features

• Bulk scraping - Process multiple Twitter users in a single run
• Timeline scraping - Extract tweets from any public X (Twitter) user timeline
• Profile URL support - Accept both usernames and full profile URLs (twitter.com/x.com)
• User information - Retrieve user profile data including verification status
• Engagement metrics - Get retweet, like, and reply counts for each tweet
• Media detection - Identify tweets with attached media (images, videos)
• Guest mode operation - Works without authentication requirements
• Batch processing - Process users in configurable batches with delays
• Rate limiting protection - Configurable delays between requests
• Structured output - Results include comprehensive tweet and user data
• Configurable limits - Control the number of tweets retrieved

Input Parameters

Field	Type	Required	Description
username	Array/String	Yes	Username(s) or profile URL(s) to scrape. Supports: elonmusk, @elonmusk, https://x.com/puma, https://twitter.com/nasa?ref=xyz
count	Number	No	Number of tweets to retrieve per user (default: 20, max: 100)
includeReplies	Boolean	No	Whether to include reply tweets in the results (default: false)
includeRetweets	Boolean	No	Whether to include retweets in the results (default: true)
batchSize	Number	No	Number of users to process in each batch (default: 5, max: 20)
delayBetweenRequests	Number	No	Delay in seconds between processing each user (default: 2.0)

Example Input - Single User

{
  "username": "elonmusk",
  "count": 10,
  "includeReplies": false,
  "includeRetweets": true
}

Example Input - Bulk Scraping

{
  "username": [
    "elonmusk",
    "https://x.com/puma",
    "@nasa",
    "https://twitter.com/apify"
  ],
  "count": 20,
  "includeReplies": false,
  "includeRetweets": true,
  "batchSize": 5,
  "delayBetweenRequests": 2.5
}

How It Works

1. The actor validates and normalizes input parameters (usernames/URLs, count, etc.)
2. Extracts usernames from various input formats (plain username, @username, or full URLs)
3. Initializes a Twitter guest client (no authentication required)
4. Processes users in batches with configurable delays to avoid rate limiting
5. For each user:
   • Looks up the user ID from the username
   • Fetches the user's timeline tweets
   • Extracts comprehensive tweet data, user information, and engagement metrics
6. Results are pushed to the dataset with structured tweet and user information

Output

The actor outputs a structured object containing:

{
  "username": "elonmusk",
  "user_id": "44196397",
  "tweet_count": 20,
  "tweets": [
    {
      "id": "1234567890",
      "text": "Tweet content here",
      "created_at": "2025-07-07T12:34:56",
      "retweet_count": 1000,
      "favorite_count": 5000,
      "reply_count": 200,
      "is_retweet": false,
      "is_reply": false,
      "user": {
        "id": "44196397",
        "name": "Elon Musk",
        "screen_name": "elonmusk",
        "verified": true
      },
      "media": [
        {
          "type": "photo",
          "url": "https://pbs.twimg.com/media/example.jpg"
        }
      ]
    }
  ]
}

Example Usage

Single User - Plain Username

{
  "username": "elonmusk",
  "count": 10
}

Single User - Profile URL

{
  "username": "https://x.com/puma?ref=homepage",
  "count": 15,
  "includeReplies": true,
  "includeRetweets": false
}

Bulk Scraping - Multiple Users

{
  "username": [
    "elonmusk",
    "https://x.com/nasa",
    "@apify",
    "https://twitter.com/google"
  ],
  "count": 20,
  "batchSize": 2,
  "delayBetweenRequests": 3.0
}

contact

email : coredev.dan@gmail.com