Twitter Profile Scraper

Twitter Profile Scraper

Twitter Profile Scraper is a fast, reliable Twitter/X profile and tweet data extractor that turns public profiles and posts into structured datasets for analysis. It solves the challenge of collecting clean, consistent timelines and profile metadata at scale by combining authorized API requests with lightweight browser automation. Built for marketers, developers, data analysts, and researchers, this Twitter profile scraping tool works as a Twitter user profile scraper and X profile scraper for bulk workflows, lead gen, and CRM enrichment—enabling automation-ready pipelines at scale.

What data / output can you get?

Below are the real fields this Twitter profile data extractor returns to the Apify dataset. Values are collected from public timelines and legacy user objects.

Data type	Description	Example value
id_str	Tweet ID (string)	"1519480761749016577"
full_text	Full tweet text	"Next I'm buying Coca-Cola to put the cocaine back in"
created_at	Tweet creation time	"Thu Apr 28 00:56:58 +0000 2022"
favorite_count	Likes count	4289223
retweet_count	Retweets count	594435
reply_count	Replies count	170050
quote_count	Quote count	167104
bookmark_count	Bookmark count (defaults to 0 if missing)	21112
conversation_id_str	Conversation/thread ID	"1519480761749016577"
user_id_str	Author user ID	"44196397"
lang	Tweet language code	"en"
is_quote_status	Whether the tweet is a quote	false
favorited	Whether the tweet is favorited by the viewer	false
retweeted	Whether the tweet is retweeted by the viewer	false
bookmarked	Whether the tweet is bookmarked by the viewer (defaults false)	false
display_text_range	Visible character range	[0, 52]
entities	Entities object with hashtags, mentions, urls, plus timestamps[]	{"hashtags": [], "urls": [], "timestamps": []}
extended_entities	Extended media entities (if available)	null
user	Legacy user profile object for the tweet author	{"screen_name": "elonmusk", "name": "Elon Musk", ...}

Notes:

• The user object contains legacy profile fields such as screen_name, name, description (bio), followers_count, profile_image_url, profile_banner_url, created_at, verified, entities, etc.
• Depending on settings, the actor can also output profile-only records or flag notFound/suspended profiles for auditability.
• Export results from the Apify dataset in JSON, CSV, or Excel to plug into your analytics, enrichment, or BI tools.

Key features

• 🚀 High-throughput tweet collection
  Scrape Twitter user profiles efficiently via the X GraphQL API with robust pagination—ideal for a bulk Twitter profile scraper.

• 🔐 Automatic authorization handling
  Uses Playwright to obtain the required authorization header from x.com at runtime—no login or cookies required.

• 🌐 Smart proxy fallback
  Built-in logic escalates from no proxy → datacenter → residential (with retries) on 401/403/429 blocks to keep runs stable.

• 🧠 Clean, legacy-style user profiles
  Each tweet can include the author’s legacy user object (screen_name, name, description, followers_count, images, verification, and more) for enriched analytics with a Twitter bio scraper approach.

• 📚 Flexible inputs (URLs, @handles, IDs, keywords)
  Accepts full profile URLs, usernames, numeric user IDs, or keyword queries—powering both a Twitter profile crawler and search-driven discovery.

• 🧵 Profile-only mode
  Enable “Fetch Only User Profile (No Tweets)” to return just the legacy profile object when timeline data isn’t needed.

• 🧩 Not found & suspended flags
  Optional toggles include records marked with notFound or suspended for transparent auditing in compliance workflows.

• 🧑‍💻 Developer-friendly actor
  Results stream live to the Apify dataset for programmatic access. Use it as a Twitter profile scraping API in Apify or integrate with Python workflows to build end-to-end automation.

• 📤 Export & integrate anywhere
  Download clean outputs in JSON, CSV, or Excel for dashboards, lead enrichment, and research pipelines.

How to use Twitter Profile Scraper - step by step

1. Create or log in to your Apify account
   Run this Twitter profile scraping tool in the cloud with built-in storage and exports.

2. Open the Twitter Profile Scraper actor
   Search for “Twitter Profile Scraper” on Apify and click Try for free.

3. Add your input data
   Paste a list of Twitter/X profile URLs, usernames, user IDs, or keywords into startUrls. Examples:

   • https://x.com/elonmusk
   • mrbeast
   • 44196397
   • tesla

4. Configure options

   • maxTweets sets how many tweets to fetch per user/keyword.
   • addUserInfo includes the author’s legacy user object in each tweet.
   • onlyUserInfo returns profile info only (no tweets).
   • addNotFoundUsersToOutput and addSuspendedUsersToOutput include flagged records for missing/suspended users.
   • proxyConfiguration lets you enable Apify Proxy with automatic fallback.

5. Start the run
   Click Start. The actor will obtain authorization automatically, handle proxy fallback if needed, and begin collecting tweets and/or profiles.

6. Monitor progress
   Items are pushed live to the dataset as they are collected. Inspect the table view or JSON to validate output.

7. Export your results
   Export the dataset in JSON, CSV, or Excel and feed it into your analytics, CRM, or research stack.

Pro Tip: Schedule this X profile scraper or trigger it via the Apify API to keep your datasets fresh for market research, lead gen, or model training.

Use cases

Use case name	Description
Marketing analytics – influencer tracking	Aggregate engagement metrics (likes, replies, quotes, bookmarks) and legacy follower counts to evaluate creator reach.
Competitive monitoring – content analysis	Compare posting cadence, language, and engagement across competitor accounts with a repeatable Twitter profile data extractor.
Lead research – bio & metadata enrichment	Pull legacy profile attributes (bio, verification, images) to enrich leads in CRM using a Twitter account scraper workflow.
Social listening – keyword timelines	Use keyword inputs to collect recent posts for sentiment analysis and topic modeling with a Twitter user data scraper.
Academic research – longitudinal datasets	Build reproducible datasets of public tweets and author metadata for behavioral or trend studies.
Developer pipelines – API & Python workflows	Integrate the Twitter profile scraping API outputs with your scripts or orchestration for dashboards and alerts.
Risk & trust – suspended/not found audit trails	Include notFound and suspended flags to create complete audit logs for compliance and monitoring.

Why choose Twitter Profile Scraper?

A production-ready X profile scraper built for precision, automation, and scale.

• ✅ Structured & accurate outputs
  Consistent JSON with tweet metrics and legacy user fields for immediate analysis.

• 📈 Scalable batch operations
  Handles lists of URLs, usernames, IDs, or keywords to scrape Twitter user profiles at volume.

• 🧑‍💻 Developer access & integrations
  Runs on Apify with live dataset outputs—easy to integrate into Python or API-driven pipelines.

• 🛡️ Ethical-by-design options
  Processes public data and provides explicit notFound/suspended flags for transparency.

• 🌐 Robust proxy & retries
  Automatic escalation from no proxy to datacenter, then residential with exponential backoff reduces blocks.

• 💸 Cost- and time-efficient vs extensions
  More reliable than browser extensions or unstable tools, with cloud infrastructure and monitoring.

This Twitter profile info scraper emphasizes reliability and clean data—ideal for teams replacing ad-hoc tools with a stable, automation-ready Twitter profile scraping service.

Is it legal / ethical to use Twitter Profile Scraper?

Yes—when done responsibly. This Twitter profile scraping tool accesses publicly available Twitter/X content and does not require login credentials.

Guidelines:

• Collect only public information and respect the platform’s Terms of Service.
• Avoid scraping private or login-protected data.
• Ensure compliance with applicable data protection laws (e.g., GDPR, CCPA).
• Use proxies and rate controls responsibly to prevent abuse.
• Consult your legal team for edge cases or jurisdiction-specific rules.

Input parameters & output format

Example JSON input

{
  "startUrls": [
    "https://x.com/elonmusk",
    "mrbeast",
    "44196397",
    "tesla"
  ],
  "maxTweets": 100,
  "sortOrder": "chronological",
  "addUserInfo": true,
  "onlyUserInfo": false,
  "addNotFoundUsersToOutput": false,
  "addSuspendedUsersToOutput": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Parameters

• startUrls (array, required): List of Twitter profile URLs (e.g., https://x.com/username), usernames (e.g., username), user IDs (e.g., 44196397), or search keywords (e.g., tesla). Supports bulk input. Default: none.
• maxTweets (integer, optional): Maximum number of tweets to fetch per user or keyword. Minimum 1, maximum 100. Default: 10.
• sortOrder (string, optional): Sort order for tweets (currently supports chronological order from API). One of: "chronological", "relevance". Default: "chronological".
• addUserInfo (boolean, optional): Include user profile data (legacy) in each tweet output. Default: true.
• onlyUserInfo (boolean, optional): If enabled, only fetch user profile information without tweets. Default: false.
• addNotFoundUsersToOutput (boolean, optional): Include users that were not found in the output with notFound flag. Default: false.
• addSuspendedUsersToOutput (boolean, optional): Include suspended users in the output with suspended flag. Default: false.
• proxyConfiguration (object, optional): Proxy configuration. By default, no proxy is used. If Twitter blocks requests, the actor will automatically fallback to datacenter proxy, then residential proxy with retries. Default: {"useApifyProxy": false}.

Example JSON output (tweet record)

{
  "user": {
    "screen_name": "elonmusk",
    "name": "Elon Musk",
    "description": "Mars & Cars, Chips & Dips",
    "followers_count": 229033543,
    "friends_count": 1226,
    "statuses_count": 89153,
    "favourites_count": 182734,
    "listed_count": 165176,
    "created_at": "Tue Jun 02 20:12:29 +0000 2009",
    "verified": true,
    "profile_image_url": "https://pbs.twimg.com/profile_images/...",
    "profile_banner_url": "https://pbs.twimg.com/profile_banners/...",
    "default_profile": false,
    "default_profile_image": false,
    "entities": {
      "description": {
        "urls": []
      }
    }
  },
  "id_str": "1519480761749016577",
  "full_text": "Next I'm buying Coca-Cola to put the cocaine back in",
  "created_at": "Thu Apr 28 00:56:58 +0000 2022",
  "favorite_count": 4289223,
  "retweet_count": 594435,
  "reply_count": 170050,
  "quote_count": 167104,
  "bookmark_count": 21112,
  "conversation_id_str": "1519480761749016577",
  "user_id_str": "44196397",
  "lang": "en",
  "is_quote_status": false,
  "favorited": false,
  "retweeted": false,
  "bookmarked": false,
  "display_text_range": [0, 52],
  "entities": {
    "hashtags": [],
    "symbols": [],
    "user_mentions": [],
    "urls": [],
    "timestamps": []
  },
  "extended_entities": null
}

Additional possible outputs (depending on settings)

Profile-only (onlyUserInfo: true)

{
  "user": {
    "screen_name": "example",
    "name": "Example User",
    "description": "Bio goes here...",
    "verified": false
  }
}

Not found user (addNotFoundUsersToOutput: true)

{
  "user": { "screen_name": "unknown_user" },
  "notFound": true
}

Suspended user (addSuspendedUsersToOutput: true)

{
  "user": { "screen_name": "suspended_user" },
  "suspended": true
}

Notes:

• If addUserInfo is false, tweet records will omit the user field.
• Some fields may be null or omitted when not present in the source (e.g., extended_entities).
• entities always includes a timestamps array (possibly empty) for consistency.

FAQ

What is a Twitter Profile Scraper?

It’s a Twitter/X profile scraping tool that extracts public tweets and legacy user profile data into structured outputs. It functions as a Twitter profile data extractor and Twitter user data scraper for analysis and automation.

Does it require login or cookies?

No. The actor automatically captures the authorization header from x.com using Playwright at runtime and does not require credentials in the input.

Can it scrape by username, URL, ID, or keyword?

Yes. Provide profile URLs, @handles, numeric user IDs, or keywords in startUrls to scrape Twitter user profiles or run search-based collections.

What tweet fields are included?

The output includes id_str, full_text, created_at, favorite_count, retweet_count, reply_count, quote_count, bookmark_count, conversation_id_str, user_id_str, lang, is_quote_status, favorited, retweeted, bookmarked, display_text_range, entities (with timestamps[]), extended_entities, and optionally the legacy user object.

Can I get just profile information without tweets?

Yes. Set onlyUserInfo to true to return only the legacy user profile (e.g., screen_name, name, description, followers_count, profile images, verified, etc.).

How does it handle users that are suspended or not found?

Enable addNotFoundUsersToOutput and addSuspendedUsersToOutput to include records flagged with notFound or suspended for those cases.

How does it deal with blocks and rate limits?

It uses a smart proxy fallback strategy. When needed, it escalates from no proxy to Apify datacenter proxy, then residential proxy with retries to keep runs stable.

How can I export the data?

All results are written to the Apify dataset. You can export to JSON, CSV, or Excel, or pull data programmatically for your Twitter profile scraping API workflows or Python pipelines.

Is there a free trial or pricing?

You can run this Twitter profile scraping service on Apify. Please check the actor’s Apify listing for current pricing and trial options.

Closing CTA / Final thoughts

Twitter Profile Scraper is built to extract clean, structured Twitter/X profile and tweet data at scale. With flexible inputs, enriched legacy user fields, smart proxy fallback, and live dataset outputs, it empowers marketers, developers, analysts, and researchers to automate social data collection. Use it as a bulk Twitter profile scraper, integrate via the Apify API or Python scripts, and start exporting reliable datasets for dashboards, enrichment, and research—faster and smarter.