Twitter/X Profile Scraper

A fast and efficient Apify Actor built with Python and Playwright to scrape Twitter/X user profile information.

🚀 Features

• Fast scraping using Playwright for reliable data extraction
• Simple input - just provide usernames (without @)
• Extracts essential profile data:
  • Name
  • Bio/Description
  • Location
• Handles dynamic content - waits for page elements to load properly
• Error handling - continues processing even if one profile fails

📋 Input

The Actor accepts a list of Twitter/X usernames:

{
  "usernames": ["LG1904", "username2", "username3"]
}

Note: Usernames should be provided without the @ symbol.

📤 Output

The Actor outputs profile data in the following format:

{
  "name": "LG",
  "bio": "مهتم بالعملات الرقمية والاستثمار | أتابع أحدث الأخبار وأفضل الفرص الاستثمارية | Focused on crypto investments and market opportunities. Snapchat : LG055555",
  "location": "الرياض"
}

Each profile is saved as a separate item in the dataset.

🛠️ Built With

• Apify SDK for Python - toolkit for building Apify Actors
• Playwright - browser automation library for reliable web scraping
• Input schema - validates and defines Actor input
• Dataset - stores structured output data

🚦 Getting Started

Prerequisites

• Python 3.8+
• Apify CLI installed
• Playwright browsers installed

Installation

1. Clone or download this Actor

2. Install dependencies:

   $pip install -r requirements.txt
   

3. Install Playwright browsers:

   $playwright install --with-deps
   

Running Locally

1. Update the input file storage/key_value_stores/default/INPUT.json:

   {
     "usernames": ["LG1904"]
   }
   

2. Run the Actor:

   $apify run
   

3. Check the results in storage/datasets/default/

📦 Deploy to Apify Platform

Option 1: Push from Local Machine

1. Log in to Apify:

   $apify login
   

   You'll need your Apify API Token

2. Deploy the Actor:

   $apify push
   

3. Find your Actor in Actors -> My Actors

Option 2: Connect Git Repository

1. Push your code to a Git repository (GitHub, GitLab, etc.)

2. Go to Actor creation page

3. Click Link Git Repository and follow the instructions

📊 How It Works

1. The Actor reads usernames from the input
2. For each username, it navigates to https://x.com/{username}
3. Waits for the page to load and dynamic content to render
4. Extracts profile information using CSS selectors:
   • Name: from div[data-testid="UserName"]
   • Bio: from div[data-testid="UserDescription"] span
   • Location: from span[data-testid="UserLocation"] span span
5. Saves the extracted data to the dataset

⚙️ Configuration

The Actor uses the following settings:

• Timeout: 60 seconds for page load
• Wait strategy: Uses load event (more reliable than networkidle for Twitter/X)
• Additional wait: 3 seconds for dynamic content to render
• User agent: Standard Chrome user agent to avoid detection

🔧 Troubleshooting

Timeout Errors

If you encounter timeout errors, the page might be loading slowly. The Actor already uses optimized timeouts, but you can adjust them in src/main.py if needed.

Missing Data

If some fields are missing (null), it might mean:

• The profile doesn't have that information
• The page structure changed (Twitter/X updates their HTML)
• The page didn't load completely

Rate Limiting

Twitter/X may rate limit requests if you scrape too many profiles too quickly. Consider adding delays between requests if needed.

📚 Resources

• Apify SDK for Python documentation
• Playwright Python documentation
• Apify Platform documentation
• Apify Academy
• Join Apify Discord community

📝 License

This Actor is provided as-is. Make sure to comply with Twitter/X Terms of Service when using this scraper.

🤝 Contributing

Feel free to submit issues or pull requests to improve this Actor!

📧 Support

For issues and questions:

• Check Apify documentation
• Join Apify Discord
• Contact Apify support through the platform

---

Note: This Actor is for educational and legitimate use cases only. Always respect website terms of service and rate limits.