Twitter Profile Scraper

Twitter Profile Scraper

The Twitter Profile Scraper is a fast, scalable Twitter/X profile data extractor that turns public profiles and posts into structured datasets for analysis. It solves the challenge of collecting consistent, high-quality Twitter user data at scale by automating extraction of tweet metadata and optional legacy profile fields. Built for marketers, developers, data analysts, and researchers, this Twitter profile scraper tool supports usernames, profile URLs, numeric user IDs, and keyword searches — enabling bulk Twitter profile scraping pipelines that power market research, lead gen, and brand monitoring.

What data / output can you get?

Below are the exact fields this X user data scraper can return. Fields reflect what’s pushed to the Apify dataset by the actor.

Data type	Description	Example value
id_str	Unique Tweet ID (string)	"1519480761749016577"
full_text	Tweet text	"Next I'm buying Coca-Cola to put the cocaine back in"
created_at	Tweet creation time (string)	"Thu Apr 28 00:56:58 +0000 2022"
favorite_count	Likes count	4289223
retweet_count	Retweets count	594435
reply_count	Replies count	170050
quote_count	Quotes count	167104
bookmark_count	Bookmarks count (defaults to 0 if missing)	21112
conversation_id_str	Conversation/thread ID	"1519480761749016577"
user_id_str	Author user ID (string)	"44196397"
lang	Language code	"en"
is_quote_status	Whether the tweet is a quote	false
favorited	Whether favorited by the logged-in account (legacy)	false
retweeted	Whether retweeted by the logged-in account (legacy)	false
bookmarked	Whether bookmarked by the logged-in account (legacy; defaults to false)	false
display_text_range	Visible text range indices	[0, 52]
entities	Entities object (hashtags, mentions, urls) with a guaranteed timestamps array	{"hashtags": [], "user_mentions": [], "urls": [], "timestamps": []}
extended_entities	Extended media entities (if available)	null
user	Legacy user profile object for the tweet author (included if addUserInfo = true)	{"screen_name": "elonmusk", "name": "Elon Musk", "...": "..."}

Notes:

• When onlyUserInfo is enabled, items contain only {"user":
• When addUserInfo is false, the user object is removed from tweet items.
• Bonus outputs: this Twitter profile metadata scraper can add notFound or suspended flags for profiles when configured.
• You can export results from the Apify dataset in JSON, CSV, or Excel to scrape Twitter profiles to CSV and other formats.

Key features

• ⚡ Robust proxy fallback & resilience
  Automatic fallback from no proxy → datacenter → residential with retries minimizes blocks and keeps runs stable in production.

• 🔎 Multi-input support (URLs, usernames, IDs, keywords)
  Feed the bulk Twitter profile scraper a mix of https://x.com/username links, plain usernames, numeric user IDs, or keyword queries — no code Twitter profile scraper setup required.

• 🧠 Real-time auth header capture
  Uses Playwright to dynamically collect the authorization header from X, enabling reliable access to public GraphQL endpoints without official API keys.

• 🧵 Tweets + optional legacy profile data
  Each tweet can include a legacy user object (addUserInfo), or you can switch to profile-only mode (onlyUserInfo) for a lean Twitter bio scraper workflow.

• 🏷️ Not-found and suspended user signaling
  Optionally include users that were not found or are suspended by adding notFound or suspended flags to the output.

• 📦 Structured output & convenient exports
  Clean, consistent fields for analytics pipelines and easy exports to JSON/CSV/Excel — ideal for Twitter profile scraping API workflows and BI.

• 🧩 Developer-friendly & Python-based
  Written in Python + Playwright + aiohttp with the Apify SDK. Easy to integrate into custom pipelines, scripts, or an X profile scraper API setup.

• 🔐 Production-ready infrastructure
  Live dataset saving, retries, and proxy rotation deliver reliability far beyond a lightweight Twitter profile scraper Chrome extension.

How to use Twitter Profile Scraper - step by step

1. Create or log in to your Apify account
   Access the actor from the Apify platform.

2. Open the Twitter Profile Scraper actor
   Look for “Twitter Profile Scraper” by scrapeflow and click Try for free.

3. Add your input in startUrls
   Paste a list of targets — profile URLs, usernames, numeric user IDs, or keyword searches. Examples:

   • https://x.com/elonmusk
   • mrbeast
   • 44196397
   • electric cars

4. Configure settings (optional)

   • maxTweets: Control per-user/keyword tweet count.
   • sortOrder: Choose chronological or relevance (the schema supports chronological from API).
   • addUserInfo / onlyUserInfo: Include legacy user data with each tweet or fetch profile-only.
   • addNotFoundUsersToOutput / addSuspendedUsersToOutput: Add notFound/suspended flags when applicable.
   • proxyConfiguration: Toggle and configure Apify Proxy; the actor falls back automatically if blocked.

5. Start the run
   The actor fetches an authorization header and begins collecting tweets or profile objects, saving results live to the dataset.

6. Monitor progress
   View logs and items as they stream into the dataset. The run automatically paginates and deduplicates where applicable.

7. Download results
   Open the run’s dataset and export to JSON, CSV, or Excel for analysis or integration.

Pro Tip: Schedule runs and connect the dataset to your data stack via the Apify API for a fully automated Twitter user data scraper pipeline.

Use cases

Use case name	Description
Marketing analytics – influencer tracking	Measure posting cadence and engagement (likes, replies, quotes, bookmarks) across target profiles to refine content strategy.
Competitive intel – profile & content audit	Extract tweet text and metadata from competitor accounts to benchmark themes, timing, and performance.
Lead gen – bio & activity signals	Enrich prospects by capturing legacy user fields and recent tweet signals for intelligent outreach.
Brand monitoring – keyword timelines	Feed keyword queries to collect public posts mentioning your brand and analyze sentiment drivers.
Research – public discourse analysis	Build datasets for academic or policy research using keyword-tuned timelines and language codes.
Data ops – API pipeline integration	Use the Python-based actor and Apify API to feed warehouses or ML training corpora programmatically.
QA/Compliance – suspended/not-found checks	Flag handle changes, removals, or suspensions by enabling notFound and suspended outputs.

Why choose Twitter Profile Scraper?

This X profile scraper is engineered for precision, automation, and reliability at scale — a production-ready alternative to fragile extensions and one-off tools.

• ✅ Accurate, structured outputs ready for analytics
• 🌍 Works on public profiles and keywords without login or official API keys
• 📈 Scales from single profiles to large batches with resilient proxy fallback
• 🧑‍💻 Developer-first: Python + Playwright + Apify SDK for easy integration
• 🛡️ Safe-by-design: public data only, optional compliance-friendly flags
• 💰 Cost-efficient: live dataset writes and smart pagination reduce overhead
• 🔗 Workflow-ready: export to JSON/CSV/Excel and connect via API to CRMs, BI, and automation tools

In short, this Twitter profile data extractor delivers consistent results where unstable alternatives struggle, making it ideal for teams that need dependable pipelines.

Is it legal / ethical to use Twitter Profile Scraper?

Yes — when done responsibly. This actor targets publicly available Twitter/X data and does not access private profiles or authenticated content.

Guidelines for compliant use:

• Scrape only publicly accessible data and avoid private or login-only pages.
• Respect Twitter/X terms and applicable laws (e.g., GDPR, CCPA) for your jurisdiction.
• Use proxies and automation ethically; avoid excessive request rates.
• Do not bypass security or attempt to access protected information.
• Consult your legal team for edge cases or regulated use.

Input parameters & output format

Example JSON input

{
  "startUrls": [
    "https://x.com/elonmusk",
    "mrbeast",
    "44196397",
    "electric cars"
  ],
  "maxTweets": 50,
  "sortOrder": "chronological",
  "addUserInfo": true,
  "onlyUserInfo": false,
  "addNotFoundUsersToOutput": true,
  "addSuspendedUsersToOutput": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Parameter reference

• startUrls (array, required)
  Description: List of Twitter profile URLs (e.g., https://x.com/username), usernames (e.g., username), user IDs (e.g., 44196397), or search keywords (e.g., tesla). Supports bulk input.
  Default: none (required)

• maxTweets (integer)
  Description: Maximum number of tweets to fetch per user or keyword. Default is 100.
  Default: 10
  Notes: Actor enforces a minimum of 10 and a maximum of 1000 internally.

• sortOrder (string)
  Description: Sort order for tweets (currently supports chronological order from API).
  Enum: chronological, relevance
  Default: chronological

• addUserInfo (boolean)
  Description: Include user profile data (legacy) in each tweet output.
  Default: true

• onlyUserInfo (boolean)
  Description: If enabled, only fetch user profile information without tweets.
  Default: false

• addNotFoundUsersToOutput (boolean)
  Description: Include users that were not found in the output with notFound flag.
  Default: false

• addSuspendedUsersToOutput (boolean)
  Description: Include suspended users in the output with suspended flag.
  Default: false

• proxyConfiguration (object)
  Description: Proxy configuration. By default, no proxy is used. If Twitter blocks requests, the actor will automatically fallback to datacenter proxy, then residential proxy with retries.
  Default (prefill): {"useApifyProxy": false}

Example JSON output (tweet item)

{
  "user": {
    "screen_name": "elonmusk",
    "name": "Elon Musk",
    "description": "Mars & Cars, Chips & Dips",
    "followers_count": 229033543,
    "friends_count": 1226,
    "statuses_count": 89153,
    "verified": true,
    "profile_image_url": "https://pbs.twimg.com/profile_images/...",
    "profile_banner_url": "https://pbs.twimg.com/profile_banners/..."
  },
  "id_str": "1519480761749016577",
  "full_text": "Next I'm buying Coca-Cola to put the cocaine back in",
  "created_at": "Thu Apr 28 00:56:58 +0000 2022",
  "favorite_count": 4289223,
  "retweet_count": 594435,
  "reply_count": 170050,
  "quote_count": 167104,
  "bookmark_count": 21112,
  "conversation_id_str": "1519480761749016577",
  "user_id_str": "44196397",
  "lang": "en",
  "is_quote_status": false,
  "favorited": false,
  "retweeted": false,
  "bookmarked": false,
  "display_text_range": [0, 52],
  "entities": {
    "hashtags": [],
    "symbols": [],
    "user_mentions": [],
    "urls": [],
    "timestamps": []
  },
  "extended_entities": null
}

Example JSON output (profile-only mode)

{
  "user": {
    "screen_name": "mrbeast",
    "name": "MrBeast",
    "description": "I want to make the world a better place",
    "followers_count": 27800000,
    "friends_count": 400,
    "statuses_count": 12000,
    "verified": true,
    "profile_image_url": "https://pbs.twimg.com/profile_images/...",
    "profile_banner_url": "https://pbs.twimg.com/profile_banners/..."
  }
}

Example JSON output (not found / suspended flags)

{
  "user": { "screen_name": "unknown_handle" },
  "notFound": true
}

{
  "user": { "screen_name": "suspended_handle" },
  "suspended": true
}

Notes:

• If addUserInfo is set to false, the "user" field is omitted from tweet items.
• When onlyUserInfo is true, the actor outputs only the "user" object or a flag (notFound/suspended) per input.
• The "entities" object always includes a "timestamps" array (empty by default).

FAQ

Do I need a Twitter API key to use this Twitter profile scraper tool?

No. The actor dynamically captures an authorization header via Playwright and uses public endpoints, so you don’t need official API credentials to run this Twitter profile scraping API workflow.

Can I scrape by keyword as well as by profile?

Yes. You can provide keywords in startUrls, and the actor will collect tweets from public search timelines. It also supports profile URLs, usernames, and numeric user IDs.

How many tweets can I extract per profile or keyword?

The input schema allows you to configure maxTweets, and the actor enforces a minimum of 10 and a maximum of 1000 per target internally.

Can I include or exclude user profile data with each tweet?

Yes. Set addUserInfo to include the legacy user object in each tweet. If you prefer profile-only output without tweets, enable onlyUserInfo.

What happens if a username is not found or is suspended?

You can enable addNotFoundUsersToOutput and/or addSuspendedUsersToOutput to include items with notFound or suspended flags, alongside the provided screen_name.

Does this work as a bulk Twitter profile scraper?

Yes. Provide multiple entries in startUrls (mixing URLs, usernames, IDs, and keywords) to run batch jobs and scrape Twitter profiles to CSV/JSON/Excel at scale.

Is proxy support available?

Yes. By default, no proxy is used. If requests are blocked, the actor automatically falls back to datacenter and then residential proxies with retries.

Can I use this with Python or automate via API?

Absolutely. The actor is built with Python and integrates seamlessly via the Apify API. It’s ideal for developers building a Twitter profile scraper Python workflow or an X profile scraper pipeline.

Closing CTA / Final thoughts

Twitter Profile Scraper is built for structured, reliable extraction of public Twitter/X tweets and legacy profile data. It combines resilient proxy handling, multi-input flexibility, and clean outputs so marketers, developers, analysts, and researchers can move from raw profiles to insights quickly. Use the Apify API to automate a no code Twitter profile scraper pipeline, export to CSV/JSON/Excel, and integrate with your analytics stack. Start extracting smarter, scalable Twitter user data today.