Twitter/X Profile Scraper & Profile Viewer (X.com)

A powerful Apify actor that scrapes Twitter/X profiles and saves comprehensive profile data to an Apify dataset. This tool serves as a fast, reliable, and scalable twitter profile scraper, x profile scraper, and x.com profile scraper. Use it to extract twitter profile info at scale, perform social media research, and export cleaned data for analytics.

Features

• ✅ Multiple Username Support: Process multiple Twitter usernames in a single run
• ✅ URL Support: Accept both usernames and full Twitter/X URLs
• ✅ Error Handling: Comprehensive error handling with detailed logging
• ✅ Data Normalization: Clean, structured output with all essential profile information
• ✅ Input Validation: Robust username and URL validation and sanitization
• 🚀 Profile Data Extractor: Extract twitter profile info including bio, image URLs, follower/following counts, and more
• 🧭 Followers/Following Coverage: Designed to support twitter followers scraper and twitter following scraper workflows
• 🖼️ Media Fields: Scrape profile picture from twitter (profile image URL) and banner URL when available
• 🧾 Twitter Metadata Extractor: Collect key metadata like verification status, account creation date, and profile location
• 📦 Flexible Export: Export twitter data to csv and JSON; integrate with databases via Apify dataset APIs
• 📈 Scalable & reliable: Built for production use with robust retries, throttling, and logging
• 🔍 Research-Friendly: Useful for twitter scraping tool for researchers and brand monitoring use-cases

Input Configuration

The actor accepts the following input parameters:

• usernames (array): List of Twitter usernames or full Twitter/X URLs to scrape
  • Example: ["realdonaldtrump", "elonmusk", "https://twitter.com/premierleague", "https://x.com/googlecalendar"]
  • Format: Usernames (alphanumeric and underscores) or full Twitter/X URLs

Optional Parameters

• maxUsernames (integer): Maximum number of usernames to process (default: 10, max: 100)
• delayBetweenRequests (integer): Delay between API requests in milliseconds (default: 2000, range: 1000-10000)

Example Input

{
  "usernames": ["realdonaldtrump", "elonmusk", "https://twitter.com/premierleague", "https://x.com/googlecalendar"],
  "maxUsernames": 5,
  "delayBetweenRequests": 3000
}

Output Data Structure

The actor outputs comprehensive Twitter profile data with the following fields:

Core Profile Information

• username: Original username requested
• user_id: Twitter user ID
• name: Display name
• screen_name: Twitter handle
• description: Bio/description
• location: User location
• created_at: Account creation date

Verification & Status

• verified: Blue checkmark status
• profile_protected: Whether account is private
• can_dm: Direct message permissions
• can_media_tag: Media tagging permissions
• has_graduated_access: Premium access status

Statistics

• followers_count: Number of followers
• friends_count: Number of following
• statuses_count: Number of tweets
• favourites_count: Number of likes
• listed_count: Number of lists
• media_count: Number of media posts

Media URLs

• profile_image_url: Profile picture URL
• profile_banner_url: Profile banner URL
• profile_image_shape: Profile image shape (Circle/Square)

Additional Data

• affiliates_highlighted_label: Business/organization labels
• verification_info: Detailed verification information
• scraped_at: Timestamp of data collection
• raw_data: Complete original API response

Basic Usage

{
  "usernames": ["realdonaldtrump", "elonmusk"]
}

URL Support

{
  "usernames": ["https://twitter.com/premierleague", "https://x.com/googlecalendar"]
}

Mixed Input (Usernames and URLs)

{
  "usernames": ["tim_cook", "https://twitter.com/premierleague", "elonmusk", "https://x.com/googlecalendar"]
}

Advanced Configuration

{
  "usernames": ["tim_cook", "https://twitter.com/premierleague", "https://x.com/googlecalendar", "sundarpichai"],
  "maxUsernames": 3,
  "delayBetweenRequests": 5000
}

Use Cases (Who is this for?)

• Twitter scraper for lead generation: Build prospect lists from public profiles
• Influencer finder on Twitter: Identify creators by bio keywords, follower counts, and verification
• Social media research scraper: Academic and market research on public profiles
• Scrape Twitter for brand monitoring: Track changes in bios, links, and verification
• Competitor analysis Twitter data: Benchmark public metrics across competitor accounts
• Extract Twitter data for academic research: Export clean datasets for reproducible studies

Export & Output Options

• Export twitter data to CSV: Download tabular datasets for spreadsheets and BI tools
• Export twitter profile data to JSON: Preserve full structure for programmatic analysis
• Save twitter profiles to database: Use Apify datasets and integrations to pipe to SQL/NoSQL
• Download twitter user list: From your run results with a single click
• Bulk export twitter profiles: Process large batches efficiently

Performance & Scalability

• Fast twitter profile scraper: Efficient batching minimizes overhead
• Reliable twitter scraper: Retries and error handling for stability
• Scalable twitter data scraper: Horizontal scaling via Apify platform
• Multi-threaded scraper for Twitter: Parallelized requests within platform limits
• Lightweight twitter scraping tool: Minimal input surface; streamlined output
• Stable for production: Logging and throttling for long-running jobs

Technical Guide for Developers

• Concepts: headless browser twitter scraping, handle twitter rate limit, rotate proxies for twitter scraping
• Patterns: pagination in twitter scraper for lists and result sets (where applicable)
• Optional approaches: selenium twitter scraper, puppeteer twitter scraper (depending on your stack)
• Twitter API alternatives: twitter api profile fetch or x api user endpoint may require credentials and are subject to ToS; this actor targets public data collection workflows.
• Operational concerns: bypass twitter login for scraping where legal and feasible; captcha solving for twitter scraper may require third-party services—evaluate compliance first.

Compliance, Ethics, and Policy

• Twitter scraping legality: Laws vary by jurisdiction. Consult legal counsel for your use case.
• Is twitter scraping allowed?: Review Twitter/X Terms of Service and policies before scraping.
• Ethical twitter scraping: Collect only public data, avoid abusive rates, and respect user privacy.
• GDPR-compliant twitter scraper: If processing personal data, ensure a lawful basis and data subject rights.
• Robots.txt & site policies: Check robots.txt twitter scraping rules and honor rate limits where applicable.
• This project does not condone: twitter scraper hack, crack twitter scraper tool, twitter password extractor, or any illegal twitter scraper usage.

Error Handling

The actor includes comprehensive error handling for:

• Rate Limiting: Automatic handling with configurable delays
• Network Issues: Connection timeouts and retries
• Invalid Usernames: Format validation and sanitization

Failed requests are logged with error details and saved to the dataset for analysis.

Performance & Rate Limiting

• Default Delay: 2 seconds between requests
• Configurable Delays: 1-10 seconds between requests
• Timeout: 30-second timeout per request
• Batch Processing: Process multiple usernames efficiently

Handling Rate Limits and Proxies

• Handle twitter rate limit: Tune delayBetweenRequests and increase backoff on repeated failures
• Rotate proxies for twitter scraping: Use Apify proxy groups or your proxies for higher throughput and resilience
• Backoff & retries: Automatic retries with conservative pacing to keep runs stable

How It Works (High-level)

1. Validates and normalizes input usernames and URLs
2. Fetches public profile data with platform-safe requests
3. Normalizes fields (IDs, counts, media URLs, verification, timestamps)
4. Saves items to an Apify dataset for easy export (CSV, JSON, API)

SEO & Discovery (Keyword Index)

Core: twitter profile scraper, x profile scraper, twitter scraper, x scraper, twitter user scraper, twitter account scraper, x.com profile scraper

Features: profile data extractor, twitter user data extractor, extract twitter profile info, scrape twitter bio, twitter followers scraper, twitter following scraper, scrape profile picture from twitter, twitter metadata extractor, scrape tweets from profile, twitter timeline scraper

Long-tail: how to scrape twitter profiles, best twitter profile scraper 2025, twitter profile scraper open source, scrape multiple twitter profiles, bulk twitter profile scraper tool, extract twitter user info from handle, export twitter profiles to csv, scrape twitter profiles without API, twitter scraping tool for researchers, how to download twitter profile data

Technical: twitter api profile fetch, x api user endpoint, headless browser twitter scraping, selenium twitter scraper, puppeteer twitter scraper, rotate proxies for twitter scraping, handle twitter rate limit, bypass twitter login for scraping, captcha solving for twitter scraper, pagination in twitter scraper

Data fields: twitter username, twitter handle, profile bio, profile image url, account creation date, follower count, following count, verified badge status, profile location, website link in profile

Use-cases: twitter scraper for lead generation, influencer finder on twitter, social media research scraper, scrape twitter for brand monitoring, competitor analysis twitter data, extract twitter data for academic research

Export: export twitter data to csv, export twitter profile data to json, save twitter profiles to database, download twitter user list, bulk export twitter profiles

Ecosystem: open source twitter scraper, no code twitter scraper, twitter scraping chrome extension, SaaS twitter data extractor, twitter scraping Python tool, twitter scraper GitHub

Performance: fast twitter profile scraper, reliable twitter scraper, scalable twitter data scraper, multi-threaded scraper for twitter, lightweight twitter scraping tool, stable twitter scraper for production

Compliance: twitter scraping legality, is twitter scraping allowed, twitter scraping policy, ethical twitter scraping, GDPR-compliant twitter scraper, twitter terms of service scraper, scraping public data from twitter, robots.txt twitter scraping rules

Best Practices

1. Request Delays: Use appropriate delays to avoid rate limiting
2. Username Validation: Ensure usernames are valid before processing
3. Error Monitoring: Monitor logs for failed requests
4. Data Storage: Use Apify datasets for reliable data storage

Dataset Export (CSV/JSON)

• From Apify Console: After a run, open the run’s Dataset and use the Export buttons for CSV or JSON.
• API (replace DATASET_ID with your dataset ID)

Direct links:

• CSV: https://api.apify.com/v2/datasets/igyf1aapEP59CadUF/items?format=csv&clean=true
• JSON: https://api.apify.com/v2/datasets/igyf1aapEP59CadUF/items?format=json&clean=true

Optional query parameters:

• attachment=1 to force download
• fields=username,screen_name,followers_count,verified to select columns
• delimiter=; for CSV delimiter

Examples:

$curl "https://api.apify.com/v2/datasets/igyf1aapEP59CadUF/items?format=csv&clean=true&attachment=1&fields=username,screen_name,followers_count,verified"

$curl "https://api.apify.com/v2/datasets/igyf1aapEP59CadUF/items?format=json&clean=true"

Tip: You can find the DATASET_ID on the run detail page or via the run’s defaultDatasetId field in the API response.

FAQ

• Can I scrape Twitter profiles without the official API?

  • Yes. This actor targets public profile data collection workflows. Always follow the platform’s Terms and applicable laws.

• Do I need to log in?

  • Not necessarily for public data. Some profiles or scenarios may require authentication; evaluate compliance and operational needs before enabling it.

• How do I handle rate limits?

  • Increase delayBetweenRequests, enable retries/backoff, and consider using Apify proxy groups to distribute traffic.

• Can I rotate proxies?

  • Yes. Use Apify proxies or your own proxy infrastructure for higher throughput and resilience.

• Does this scrape tweets/timelines?

  • This actor focuses on profile metadata (bio, counts, images, verification, etc.). Timeline scraping requires a separate workflow.

• What export formats are supported?

  • CSV and JSON via Dataset exports. You can also integrate the dataset with databases and external tools.

• How do I run this in bulk?

  • Provide a large usernames list and tune maxUsernames and delayBetweenRequests. Consider splitting very large jobs into multiple runs.

• What about private or suspended accounts?

  • Private/suspended/removed profiles may return limited or no data. These events are logged for review.

• Is scraping allowed/legal?

  • Laws and policies vary. Consult legal counsel, respect robots.txt and Terms, and collect only public data responsibly.

Support

For issues and questions:

• Check the Apify logs for detailed error messages
• Ensure usernames are valid and public

License

This actor is provided as-is for educational and commercial use.