Twitter / X Profile Scraper

🐦 Twitter / X Profile Scraper is a powerful Apify Actor designed to discover, track, and collect public Twitter / X Profile data directly from X's own profile pages using a real headless browser. This tool provides comprehensive Twitter / X Profile intelligence including full name, bio/description, follower count, following count, and profile picture for any accessible Twitter / X Profile. Whether you're building a social media dashboard, tracking competitor accounts, or conducting audience research, the Twitter / X Profile Scraper delivers actionable Twitter / X Profile insights efficiently.

With -powered rendering, OpenGraph/meta-tag extraction, in-page DOM fallback search, and reliable Apify Dataset delivery, the Twitter / X Profile Scraper ensures comprehensive Twitter / X Profile coverage across any list of usernames or profile links. It focuses on key Twitter / X Profile signals including followers, following, and bio, making it an essential tool for social research and account intelligence.

---

📋 Table of Contents

• Features
• Data Source
• How It Works
• Input
• Output
• Technical Stack
• Data Fields
• Use Cases
• Quick Start
• Configuration
• Performance
• Important Notes
• License & Legal

---

🔥 Features

• Real Browser Rendering – Uses (Chromium) to load each Twitter / X Profile page exactly as a real visitor would see it.
• Flexible Input Handling – Accepts plain usernames, @handles, or full Twitter / X Profile URLs and normalizes them automatically.
• OpenGraph Meta Extraction – Pulls full name, description, and profile picture directly from X's og:title, og:description, and og:image meta tags.
• Regex Follower Parsing – Extracts follower and following counts from the meta description using targeted pattern matching.
• In-Page DOM Fallback – Automatically searches rendered follower/following links on the page when meta data is incomplete.
• 404 Detection – Automatically detects and skips Twitter / X Profile pages that no longer exist.
• Residential Proxy Support – Apify residential proxy wired directly into for reliable Twitter / X Profile access.
• Batch Processing – Scrape any number of Twitter / X Profile entries in a single run.
• Human-Like Pacing – Built-in delay between Twitter / X Profile requests to reduce blocking.
• Real-Time Dataset Push – Pushes all collected Twitter / X Profile records to Apify Dataset in a single batch.
• Graceful Error Handling – Skips a failing Twitter / X Profile without stopping the whole run.

---

📊 Data Source

X (Twitter) Public Profile Pages

• Authority: X's own public-facing profile pages
• Access Method: Headless Chromium via , rendering x.com/{username}
• Coverage: Any public Twitter / X Profile reachable without login
• Data: Rendered OpenGraph meta tags and in-page follower/following DOM elements
• Access: Public pages, no API key required
• Update Frequency: Reflects each Twitter / X Profile's current live state at time of run

---

⚙️ How It Works

The Twitter / X Profile Scraper accepts a list of usernames or profile URLs, normalizes each into a clean username, and loads the profile page in a real headless browser session. It parses the rendered page's OpenGraph meta tags to extract full name, description, and profile picture. Follower and following counts are first parsed from the meta description using regex; if either is still missing, the Actor evaluates the live DOM to read follower/following links directly off the page. Every successfully scraped Twitter / X Profile record is collected and pushed to the Apify Dataset in a single batch at the end of the run.

Key Processing Steps:

1. Input Parsing – Accept a list of usernames, @handles, or full Twitter / X Profile URLs
2. Username Normalization – Strip @ symbols and extract the username from full URLs
3. Proxy Setup – Configure Apify residential proxy credentials for
4. Browser Launch – Start a headless Chromium instance with a realistic user agent
5. Profile Loop – Iterate through each normalized Twitter / X Profile username
6. Page Navigation – Load x.com/{username} and check for 404s
7. Meta Tag Parsing – Extract full name, description, and profile picture from OpenGraph tags
8. Regex Extraction – Parse follower/following counts from the meta description
9. DOM Fallback – Read follower/following counts directly from rendered page links if still missing
10. Timestamp Tagging – Record scrape time per entry
11. Rate Limiting – Apply a short delay between Twitter / X Profile requests
12. Dataset Push – Push all collected records from the run in a single batch

Key Benefits:

• Pull Twitter / X Profile data for any public account without manual visits
• Detect missing or suspended Twitter / X Profile pages automatically
• Monitor creator and brand accounts' Twitter / X Profile metrics over time
• Feed social dashboards, research pipelines, or influencer-tracking tools
• Automate recurring Twitter / X Profile checks as follower counts change

---

📥 Input

The Actor accepts the following input parameters:

Field	Type	Default	Description
profiles	array	(required)	List of Twitter / X Profile usernames, @handles, or full profile URLs.
useApifyProxy	boolean	true	Enable Apify residential proxies.
apifyProxyGroups	array	["RESIDENTIAL"]	Proxy group configuration.

Example Input:

{
  "profiles": [
    "elonmusk",
    "@nasa",
    "https://x.com/natgeo"
  ],
  "useApifyProxy": true,
  "apifyProxyGroups": ["RESIDENTIAL"]
}

Scrape a Single Twitter / X Profile:

{
  "profiles": ["natgeo"]
}

Scrape from Full Profile URLs:

{
  "profiles": [
    "https://x.com/nasa",
    "https://x.com/natgeo"
  ]
}

---

📤 Output

The Actor pushes Twitter / X Profile records with the following structure:

Field	Type	Description
username	string	The normalized Twitter / X Profile username used for the request
full_name	string	Display name extracted from the og:title meta tag
profile_url	string	The full Twitter / X Profile URL
description	string	Profile bio/description text
followers	string/null	Follower count as displayed on the Twitter / X Profile (e.g., "1.2M Followers")
following	string/null	Following count as displayed on the Twitter / X Profile (e.g., "342 Following")
profile_pic	string	URL of the Twitter / X Profile picture
scraped_at	string	ISO 8601 timestamp of when the record was scraped

Example Twitter / X Profile Record:

{
  "username": "natgeo",
  "full_name": "National Geographic",
  "profile_url": "https://x.com/natgeo",
  "description": "Taking the world's best photos daily since 1888.",
  "followers": "23.1M Followers",
  "following": "312 Following",
  "profile_pic": "https://pbs.twimg.com/profile_images/...",
  "scraped_at": "2026-08-06T09:00:00+00:00"
}

---

🧰 Technical Stack

• Browser Automation: `` (Chromium) for real, rendered access to Twitter / X Profile pages
• HTML Parsing: BeautifulSoup for OpenGraph and meta tag extraction
• Pattern Matching: re for follower/following count extraction from meta descriptions
• Date Handling: datetime with timezone-aware UTC timestamps
• Async: asyncio for non-blocking, multi-profile processing
• Proxy: Apify Proxy with RESIDENTIAL configuration, wired into 's launch options
• Logging: Apify Actor logging system
• Platform: Apify Actor serverless environment

---

📊 Data Fields Explained

Identity

• username: The normalized username used to build the Twitter / X Profile URL
• full_name: The public display name shown on the Twitter / X Profile

Profile Details

• description: The account's bio/description text
• profile_url: The canonical link to the Twitter / X Profile
• profile_pic: Direct link to the profile's avatar image

Engagement Metrics

• followers: Follower count as publicly displayed on the Twitter / X Profile
• following: Following count as publicly displayed on the Twitter / X Profile

---

🎯 Use Cases

• Creator Research – Evaluate follower and engagement metrics before an influencer partnership
• Competitor Tracking – Monitor a competitor's Twitter / X Profile growth over time
• Content Auditing – Verify profile bios and branding are up to date
• Marketing Analytics – Feed Twitter / X Profile metrics into campaign dashboards
• Media Monitoring – Track how partner or affiliate accounts present themselves
• Academic Research – Study public Twitter / X Profile trends at scale
• OSINT Research – Gather publicly available Twitter / X Profile metadata for investigations

---

🚀 Quick Start

1. Prepare Input

Go to Apify Console and enter:

{
  "profiles": ["nasa", "natgeo"]
}

2. Run the Actor

Click Start. The Actor will:

• Normalize each username or URL into a clean Twitter / X Profile username
• Launch a headless browser and load each profile page
• Extract meta data and fall back to DOM parsing as needed
• Push all results to the Dataset

3. Monitor Progress

Console shows:

Initializing Twitter Scraper for 2 account(s)...
[Twitter Scraper] Navigating to profile: '@nasa'
[Twitter Scraper] Navigating to profile: '@natgeo'
Done! Successfully saved 2 Twitter profile(s) to Dataset.

4. View & Download Results

• Results Tab: All Twitter / X Profile records
• Export: JSON, CSV, Excel, or HTML
• Filter: By follower count or username
• API Access: Available via the Apify API

---

⚙️ Configuration

Single Twitter / X Profile:

{
  "profiles": ["natgeo"]
}

Mixed usernames and URLs:

{
  "profiles": [
    "@nasa",
    "https://x.com/natgeo"
  ]
}

Run without proxy:

{
  "profiles": ["natgeo"],
  "useApifyProxy": false
}

---

📈 Performance

Processing Speed

• One page load per Twitter / X Profile, plus in-page DOM evaluation when needed
• 2 second delay between profiles to reduce blocking
• Slower than raw HTTP scraping since a full browser is launched, but far more resilient to X's client-rendered pages

Resource Usage

• Memory: Moderate, since a Chromium browser instance is launched
• Network: One page navigation per Twitter / X Profile
• Proxy: One residential proxy tunnel shared across the browser context for the run

---

⚠️ Important Notes

Legal & Compliance

• Fair Use: Respects X's public pages and applies pacing delays between requests
• Public Data Only: Retrieves only publicly visible Twitter / X Profile data exposed on the rendered page
• Privacy: Collecting profile data may be subject to privacy regulations and X's Terms of Service — use only for legitimate purposes
• Legal: Not legal advice — consult qualified professionals before using this data for compliance-sensitive decisions

Data Quality

• Freshness: Reflects each Twitter / X Profile's live state at time of run
• Completeness: Follower/following counts depend on which extraction layer succeeds; some profiles may return partial data
• Accuracy: Sourced directly from X's own rendered pages
• Verification: Cross-check high-stakes figures against the live X app or website

Best Practices

• Always configure a residential proxy for the most reliable Twitter / X Profile access
• Use full profile URLs when a username alone is ambiguous
• Re-run periodically to track changes in follower or following counts
• Expect followers or following to occasionally be null if all extraction layers fail
• Batch multiple usernames together in one run to reduce overhead

---

📄 License & Legal

Terms of Use:

• Use for legitimate research, marketing, and analytics purposes
• Respect X's Terms of Service
• Do not use for unsolicited contact, harassment, or unlawful surveillance
• Use Twitter / X Profile data responsibly and in compliance with applicable privacy laws

Disclaimer: Twitter / X Profile Scraper is provided as-is for research and analytics purposes. Users are responsible for compliance with X's ToS and all applicable laws. This is not legal advice.

---

⚖️ Twitter / X Profile Excellence

This Actor is optimized for Twitter / X Profile research with:

• ✅ Real, browser-rendered data accuracy
• ✅ Multi-layer fallback extraction (meta tags → regex → DOM search)
• ✅ Flexible username/URL input handling
• ✅ 404 and missing-profile detection
• ✅ Real-time Dataset push
• ✅ Production-ready code