Twitter Scraper

🐦 Twitter Profile Scraper – Extract X Account Data, Bios & Follower Counts

This Twitter profile scraper extracts public account-level data from X (formerly Twitter) using nothing more than a username or a profile URL. Point it at a list of handles and it returns a clean, flat JSON record for each account: display name, bio, location, follower and following counts, total posts, media count, verification status, account creation date, avatar and header image URLs, and the numeric account ID that X uses internally.

To be explicit about scope, because it matters when you are choosing a tool: this is a profile scraper, not a timeline or search scraper. It queries X's UserByScreenName GraphQL endpoint and returns the account object behind a profile page. It does not return individual tweets, replies, threads, timelines, search results, or follower lists. If you need account intelligence — audience size, bio text, verification tier, account age, posting volume — this is the right tool. If you need post text, it is not.

The actor is built in Python with curl_cffi browser impersonation and runs headlessly, so there is no browser overhead and no rendering cost. Guest-token authentication is handled internally, refreshed automatically when X rate-limits a request, and paired with rotating residential proxies. Feed it ten handles or ten thousand; the workflow is identical.

---

📊 What Data Can You Extract with This Twitter Profile Scraper?

Every successful run pushes one dataset item per X account. Fields are grouped below by what they tell you.

Category	Fields	What You Get
🆔 Account identity	profile, name, id, rest_id, status	The @handle, the human-readable display name, and X's stable internal account ID. status marks the record state.
✅ Verification & affiliation	blue_verified, verification_type, affiliates, business_account	Whether the account carries a blue check, which verification tier it sits in, any affiliated parent organisation label, and business-account metadata.
📝 Bio & location	desc, location	Full profile description text and the self-declared location string exactly as the account owner typed it.
👥 Audience metrics	sub_count, friends	Follower count (sub_count) and the number of accounts this profile follows (friends) — the two numbers behind every engagement ratio.
📈 Activity metrics	statuses_count, media_count, pinned_tweet_ids_str	Lifetime post count, lifetime media uploads, and the ID string of any pinned post on the profile.
🖼️ Visual branding	avatar, header_image	Direct HTTPS URLs to the profile picture and the profile banner image.
🔒 Account age & privacy	created_at, protected	The account registration timestamp and a boolean flag for whether the account is protected (private).

The genuinely useful pairing here is sub_count against friends. A profile following 40,000 accounts to hold 41,000 followers behaves very differently from one following 300 to hold 41,000 — the follower-to-following ratio separates organically grown audiences from follow-back inflation, and this Twitter profile scraper hands you both numbers in the same record so the calculation is a single division.

---

🌟 Key Features of the Twitter Profile Scraper

Feature	Description
🎯 Two ways to supply input	Pass bare usernames via twitterHandles or paste full profile URLs via startUrls. The actor strips the handle out of each URL automatically.
🔁 Automatic deduplication	Handles appearing in both inputs, or repeated within one input, are collapsed to a single scrape — you are never billed twice for the same account.
🛡️ Guest-token authentication	The actor activates and refreshes X guest tokens on its own. No login, no cookies, no session file to maintain.
🌐 Residential proxy rotation	When Apify Proxy is enabled, a fresh residential IP is drawn for every request attempt, which is what keeps large handle lists from stalling.
♻️ Five-attempt retry logic	HTTP 429, 401 and 403 responses trigger a forced token refresh and an escalating backoff (3s, 6s, 9s…) before the next attempt.
🧬 Browser fingerprint impersonation	curl_cffi mimics a real Chrome TLS handshake, so requests do not look like a scripted HTTP client at the transport layer.
⚡ No browser required	Direct GraphQL calls instead of Playwright or Puppeteer means faster runs and lower compute usage per profile.
🧹 Flat, analysis-ready output	Twenty top-level fields, no deep nesting to unpack. Export straight to CSV or Excel without a flattening step.
🚦 Graceful handling of dead accounts	Suspended, deleted and unavailable profiles are logged and skipped rather than crashing the run or polluting the dataset.

---

🚀 Why Choose This Twitter Profile Scraper?

Honest scope. This actor does one job and states it plainly. It returns X account profiles — not tweets, not timelines, not search results. You will not discover halfway through a project that the field you needed was never in the schema, and the input form does not advertise options the code ignores.

No credentials, no risk to your account. Authentication runs entirely on X's public guest-token flow. You never hand over a username, password, cookie jar or API bearer key, which means there is no account of yours that can be rate-limited, flagged or suspended as a result of a scrape.

Built for lists, not single lookups. The input accepts arrays. Drop in a competitor set, a creator shortlist, an influencer database or an entire CRM export of X handles and the actor iterates through them sequentially with per-profile logging, so a single bad handle never takes down the batch.

Resilient by design. Rate limiting is the normal failure mode when scraping X, not the exception. Rather than failing fast, the actor forces a new guest token, rotates to a new residential IP and backs off progressively across five attempts — behaviour that turns transient blocks into a few extra seconds instead of a lost run.

---

📥 Input

The Twitter profile scraper takes three inputs. All of them are optional, but you must supply at least one handle through either twitterHandles or startUrls or the run will finish immediately with an empty dataset.

{
  "twitterHandles": [
    "elonmusk",
    "nasa",
    "apify"
  ],
  "startUrls": [
    { "url": "https://x.com/OpenAI" },
    { "url": "https://twitter.com/github" }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

🔧 Twitter Profile Scraper Input Fields

Field	Type	Required	Default	Description
twitterHandles	array	No	["elonmusk"] (prefill)	X (Twitter) usernames to scrape, without the @ symbol. Whitespace is trimmed and duplicates removed.
startUrls	array	No	—	Full X (Twitter) profile URLs. The username is taken from each URL. Accepts both x.com and twitter.com links, plain strings or { "url": "…" } objects.
proxyConfiguration	object	No	{ "useApifyProxy": true }	Proxy settings for the run. Recommended to avoid rate limiting. When useApifyProxy is true, the actor uses Apify residential proxies with per-attempt IP rotation.

💡 Input Examples

Handles only — the simplest possible run:

{
  "twitterHandles": ["nasa", "esa", "isro"],
  "proxyConfiguration": { "useApifyProxy": true }
}

Profile URLs pasted straight from the browser:

{
  "startUrls": [
    { "url": "https://x.com/Apify" },
    { "url": "https://twitter.com/ApifyOffice/" }
  ],
  "proxyConfiguration": { "useApifyProxy": true }
}

Mixed input for a competitor benchmarking batch:

{
  "twitterHandles": ["stripe", "shopify"],
  "startUrls": [{ "url": "https://x.com/square" }],
  "proxyConfiguration": { "useApifyProxy": true }
}

---

📤 Output

Each scraped account becomes one dataset item with the structure below. Values shown are illustrative of the shape and types you should expect.

{
  "status": "active",
  "profile": "apify",
  "rest_id": "1259243184",
  "blue_verified": true,
  "verification_type": "Business",
  "affiliates": {},
  "business_account": {},
  "avatar": "https://pbs.twimg.com/profile_images/1234567890/avatar_normal.jpg",
  "header_image": "https://pbs.twimg.com/profile_banners/1259243184/1600000000",
  "desc": "Full-stack web scraping and browser automation platform.",
  "name": "Apify",
  "protected": false,
  "location": "Prague, Czechia",
  "friends": 812,
  "sub_count": 15420,
  "statuses_count": 6743,
  "media_count": 1129,
  "pinned_tweet_ids_str": ["1700000000000000000"],
  "created_at": "Tue Mar 12 09:41:22 +0000 2013",
  "id": "1259243184"
}

🧾 Twitter Profile Scraper Output Fields

Field	Type	Description
status	string / null	Status of the item. Set to active for a successfully scraped profile.
profile	string / null	The account's screen name (handle), without @.
rest_id	string / null	X's internal REST identifier for the account.
blue_verified	boolean	Whether the account carries the blue verification badge.
verification_type	string / null	Verification tier, where one applies (for example a business or government label). Null for standard accounts.
affiliates	object / null	Affiliated-organisation highlight label attached to the profile. Empty object when none.
business_account	object / null	Business account metadata returned by X. Empty object when the account is not a business account.
avatar	string	Direct HTTPS URL of the profile picture.
header_image	string	Direct HTTPS URL of the profile banner image.
desc	string	The profile bio text.
name	string / null	Display name shown above the handle.
protected	boolean	true when the account is private and its posts are restricted to approved followers.
location	string	Self-reported location string from the profile. Free text — may be empty, a city, a country or a joke.
friends	integer	Number of accounts this profile follows.
sub_count	integer / null	Follower count for the account.
statuses_count	integer / null	Lifetime number of posts made by the account.
media_count	integer / null	Lifetime number of media items (images and videos) posted.
pinned_tweet_ids_str	array	ID string(s) of the post pinned to the top of the profile. Empty when nothing is pinned.
created_at	string / null	Account registration timestamp in X's native format.
id	string / integer	Unique identifier for the account, mirroring rest_id.

---

💻 How to Use the Twitter Profile Scraper (Step by Step)

Step 1: Open the Actor and Prepare Your Handle List

Start on the actor page in Apify Console and open the Input tab. Before you touch anything, collect the X usernames you want. They can come from a spreadsheet column, a CRM export, a list of conference speakers, or simply copied from browser tabs. Strip any @ symbols — the scraper expects bare usernames such as nasa, not @nasa. If your list is already in URL form, that is fine too; you will use a different field for it in the next step.

Step 2: Enter Usernames or Profile URLs

Paste bare usernames one per line into Twitter Handles. If your source material is full profile links, paste them into Profile URLs instead — the actor takes the last path segment of each URL, drops any query string, and uses that as the handle. Both https://x.com/name and https://twitter.com/name are recognised, with or without a trailing slash. You can populate both fields in the same run; duplicates across them are removed before scraping begins, so overlap costs you nothing.

Step 3: Enable Proxy Configuration

Leave Proxy Configuration set to use Apify Proxy. This is the single most important setting for run reliability. With it enabled, the actor requests a fresh residential IP for every attempt on every profile, which spreads your requests across many exit nodes and keeps X's rate limiter from associating them. Running without a proxy is possible for a handful of handles from a clean IP, but on lists of any real size you will start collecting 429 responses quickly.

Step 4: Start the Run and Watch the Log

Click Start. The log opens immediately and narrates each step: how many unique profiles were queued, which handle is currently being scraped, and a success or failure line for each. Warnings such as Profile not found, is suspended or Blocked (429)… retrying are informational — the run continues. A profile that fails all five attempts is logged and skipped, leaving the rest of your batch unaffected.

Step 5: Review the Dataset

When the run finishes, open the Dataset tab. Each row is one X account with the twenty fields documented above. Use the table preview to sanity-check a few records: confirm profile matches what you asked for, that sub_count looks right for accounts you know, and that desc contains real bio text. If a handle you supplied is missing, search the log for it — the reason will be there, usually a rename, a suspension or a typo.

Step 6: Export in Your Preferred Format

Export from the Dataset tab as JSON, CSV, Excel, XML, HTML or RSS. Because the output is flat, CSV and Excel exports are immediately usable in a spreadsheet with no unnesting or JSON parsing. For analysts working in pandas or R, the JSON export loads into a dataframe in one line. Field selection is available at export time if you only need a subset such as profile, sub_count and created_at.

Step 7: Schedule or Automate for Ongoing Tracking

Profile metrics are time series, and a single snapshot tells you far less than a weekly one. Use Apify Schedules to re-run the same input daily, weekly or monthly, and each run stores its own dataset so you can compare sub_count and statuses_count over time. Attach a webhook to push finished runs into your warehouse, or connect the actor to a downstream workflow through the integrations described in the next section.

---

🔌 API Access & Integrations

Run the Twitter profile scraper from any language or platform that can make an HTTP request. The synchronous endpoint below starts a run and returns the dataset items in the same response.

curl -X POST "https://api.apify.com/v2/acts/scrapers-hub~twitter-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "twitterHandles": ["nasa", "esa"],
    "startUrls": [{ "url": "https://x.com/Apify" }],
    "proxyConfiguration": { "useApifyProxy": true }
  }'

The same run in Python with the official client:

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")

run_input = {
    "twitterHandles": ["nasa", "esa", "isro"],
    "startUrls": [{"url": "https://x.com/Apify"}],
    "proxyConfiguration": {"useApifyProxy": True},
}

run = client.actor("scrapers-hub/twitter-scraper").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["profile"], item["name"], item["sub_count"], item["statuses_count"])

Beyond the API, the actor plugs into Zapier, Make, Google Sheets, Slack and custom webhooks through Apify's integrations, so finished runs can append rows to a sheet, post a follower-count digest to a channel, or trigger any downstream service automatically.

---

💡 Best Use Cases for Twitter Profile Data

🔍 Influencer Vetting and Audience Verification

Before paying for a sponsored post, check the fundamentals. sub_count against friends exposes follow-back inflation, created_at reveals whether a "established creator" registered three months ago, and statuses_count divided by account age gives a posting cadence that is hard to fake. A profile with 200,000 followers, 90 lifetime posts and a six-month-old created_at is a very different proposition from one with the same reach built over eight years.

🏢 Competitor Benchmarking on X

Track a competitor set on a schedule and you build a longitudinal picture of their X presence. Weekly deltas in sub_count show whose audience is actually growing, statuses_count deltas show who is publishing more, and media_count deltas reveal a shift toward visual content. Changes in desc often signal repositioning — a bio rewrite usually precedes or accompanies a campaign.

🎯 Lead Enrichment and B2B Prospecting

Append X account context to CRM records you already hold. desc frequently contains a job title, company and contact route; location narrows territory assignment; business_account and verification_type help distinguish company accounts from personal ones. Enriching a prospect list with sub_count also lets sales teams prioritise contacts with genuine reach.

🛡️ Brand Protection and Impersonation Monitoring

Impersonation accounts copy the name, avatar and desc of the real brand but cannot copy its created_at or rest_id. Scrape the handles surrounding your brand name and compare: a recently created account with your display name, a near-identical header_image, blue_verified set to false and a low sub_count is a textbook impersonation candidate worth reporting.

📊 Social Media Research and Academic Datasets

Researchers studying platform dynamics need account-level metadata at scale, and this Twitter profile scraper produces exactly that shape of data. created_at supports cohort analysis by registration wave, protected quantifies privacy behaviour across a sample, and blue_verified plus verification_type allow verification tiers to be studied as a variable rather than assumed.

📈 Follower Growth Tracking and Campaign Measurement

Snapshot your own accounts and your campaign partners before, during and after a push. Because sub_count, statuses_count and media_count are all lifetime counters, differences between scheduled runs give you clean daily or weekly deltas without needing platform analytics access — useful when you are measuring accounts you do not own.

🗂️ Building and Cleaning Handle Databases

If you maintain a directory of X accounts — creators, journalists, portfolio companies, association members — periodic re-scraping keeps it honest. Handles that stop returning data have been renamed, deleted or suspended; protected flipping to true means the account went private; a changed name alongside an unchanged rest_id tells you the same account simply rebranded.

---

⚙️ Tips for Better Twitter Profile Scraping Results

• Always keep Apify Proxy enabled. Residential IP rotation is the difference between a run that completes and a run that spends its time in backoff. The default is on for a reason; turning it off is only sensible for a two- or three-handle test.
• Verify handles before you submit them. The scraper resolves accounts by username, and X usernames change. A handle that returns nothing is far more often a rename or a typo than a scraper failure — check the live profile URL in a browser first.
• Split very large lists across multiple runs. Profiles are fetched sequentially, so a list of several thousand handles is better run as several batches. Smaller batches also make failures easier to isolate and re-run.
• Prefer twitterHandles when you have clean usernames. URL parsing is reliable but adds a failure surface: links with tracking parameters, /status/ paths or trailing segments can yield the wrong handle. Bare usernames remove the ambiguity entirely.
• Schedule repeat runs if you care about trends. A one-off scrape gives you a number; a weekly schedule gives you a trajectory. Follower counts only become insightful once you can difference them.
• Read the run log before assuming a bug. Every skipped profile is logged with a reason — not found, suspended, protected or blocked after retries. That one line usually answers the question completely.

---

🛠️ Troubleshooting

The run finished instantly with an empty dataset. What happened? No usable usernames were parsed from your input. The log will say No usernames provided in input. Check that twitterHandles actually contains entries, or that your startUrls are genuine X profile links containing x.com/ or twitter.com/ — the actor ignores URLs that do not match either pattern.

Some profiles are missing from the output even though the run succeeded. Individual profiles are skipped rather than failing the run. Search the log for the handle: Profile not found means the username does not resolve (renamed, deleted, or mistyped), while a message about the account being suspended or unavailable means X itself is refusing to serve the profile object.

I keep seeing Blocked (429) warnings in the log. That is X rate-limiting the request, and the actor is designed to absorb it — it forces a fresh guest token, rotates to a new residential IP and retries with an increasing delay. If these warnings dominate the whole run, confirm that proxyConfiguration.useApifyProxy is set to true and reduce your batch size.

A private account returned no data at all. Protected accounts are frequently unavailable through the guest-token endpoint, and when X returns an unavailable result the profile is logged and skipped. Where a protected account does return data, protected will be true — but you should expect gaps in coverage for private profiles rather than complete records.

Fields like affiliates, business_account or verification_type are empty or null. This is normal and not an error. Those fields only carry values for accounts that actually have an affiliation label, a business account configuration or a specific verification tier. Ordinary personal accounts return empty objects and nulls for them.

---

❓ Frequently Asked Questions About Twitter Profile Scraping

Does this scraper return tweets? No. This is a profile scraper and it returns account-level data only — bio, follower and following counts, post and media counts, verification status, images, location and creation date. It does not return tweet text, timelines, replies, threads, quotes, likes or any search results. The only tweet-related field is pinned_tweet_ids_str, which contains the ID of a pinned post, not its content.

What exactly is a Twitter profile scraper? It is a tool that reads the public account information displayed on an X profile page and converts it into structured data. Instead of opening each profile in a browser and copying numbers by hand, you supply a list of usernames and receive a dataset with one row per account.

Do I need a Twitter or X API key to use this? No. The actor authenticates using X's public guest-token flow, which it activates and refreshes on its own. You do not need an X developer account, an API key, a bearer token or any login credentials.

Can I scrape private or protected accounts? Only partially, and often not at all. When X marks a profile as unavailable, the actor logs it and moves on. Where a protected account does return an object, the protected field will be true and you will see its public metadata, but private accounts should be treated as unreliable coverage.

How many X profiles can I scrape in one run? There is no hard limit built into the actor — it processes whatever unique handles it parses from your input. Profiles are fetched sequentially, so runtime scales roughly linearly with list size. For very large lists, splitting the work across several runs is more practical.

Can I use profile URLs instead of usernames? Yes. Put full profile links in startUrls and the actor extracts the username from the final path segment of each URL, stripping any query string. Both x.com and twitter.com domains are supported, and you can mix URLs and bare handles in the same run.

Is scraping public X profiles legal? Scraping publicly accessible data is broadly permitted in many jurisdictions, but legality depends on your location, your purpose and how you handle the data afterwards. Public profile information can still constitute personal data under GDPR and similar regimes. Take your own legal advice before running this at scale.

Why do I need a proxy for Twitter profile scraping? X rate-limits by IP address. Without rotation, a sequence of requests from one address hits limits quickly and the run slows to a crawl of retries. Apify residential proxies distribute requests across many IPs, which is what makes larger batches practical.

What does the sub_count field mean? It is the account's follower count. The actor reads X's normal followers count, falling back to the standard followers count where the former is unavailable. Pair it with friends, the number of accounts the profile follows, to compute follower-to-following ratios.

Can I get a list of an account's followers? No. Follower and following lists are not part of the output. You receive the counts (sub_count and friends) but not the identities of the accounts behind those numbers.

How current is the profile data? It is fetched live at run time, so every record reflects the account as X served it during that run. Follower counts and post counts move constantly, which is why scheduling repeat runs is the recommended approach for anything trend-related.

What happens if a username has been changed? The scrape returns nothing for the old handle and logs Profile not found. Usernames are mutable on X; rest_id and id are not. If you are maintaining a long-lived database, key it on rest_id rather than on the handle.

Can I export the results to CSV or Google Sheets? Yes. Apify datasets export to JSON, CSV, Excel, XML, HTML and RSS from the Console or the API, and the Google Sheets integration can write results directly into a spreadsheet. The flat output structure means no post-processing is needed before import.

Does this actor use a headless browser? No. It calls X's GraphQL endpoint directly over HTTP with curl_cffi browser impersonation, which reproduces a real Chrome TLS fingerprint without the memory and time cost of running an actual browser.

Can I schedule this Twitter profile scraper to run automatically? Yes. Apify Schedules run the actor on any cron expression with a saved input, and each run produces its own dataset. Combine that with a webhook to push each finished run into your own database or reporting stack.

---

🆘 Support & Feedback

Found a bug, hit an edge case, or seeing output that does not match this documentation? Open a ticket on the Issues tab of the actor page — it is the fastest route, because issues raised there arrive with your run ID attached and can be reproduced directly.

Need something this actor does not do? Custom scraping work, modified output schemas, private actors and integration help are all available. Email scraperhubapi@gmail.com with a description of what you need and an example of the data you expect.

If this Twitter profile scraper saved you time, please leave a rating and a short review on the actor page. Reviews genuinely shape which improvements get built next, and they help other people find a tool that does what it says.

---

⚖️ Disclaimer

This Twitter profile scraper collects only publicly available information from X (Twitter) — data that any visitor can see on a profile page without logging in. It does not access private accounts' restricted content, direct messages, protected posts, or anything behind authentication, and it does not require or use your X credentials.

You are responsible for how you use the data you collect. That includes ensuring your purpose is lawful in your jurisdiction, that your storage and retention practices are appropriate, and that you comply with X's Terms of Service and any contractual obligations you are under.

X profile data frequently constitutes personal data. Where GDPR, UK GDPR, CCPA or comparable privacy legislation applies, you act as the data controller for anything you collect: you need a lawful basis for processing, you must honour data subject rights, and you must apply data minimisation — collect only the fields your use case actually requires. Automated decision-making about individuals based on scraped profile metrics carries additional obligations.

Nothing in this document is legal advice. If your intended use is commercial, large-scale, or involves profiling individuals, consult a qualified lawyer before proceeding.

If you are the owner of an account that appears in data collected with this actor and you would like that data removed, email scraperhubapi@gmail.com and we will action the request.