Twitter Profile Scraper

Twitter Profile Scraper

The Twitter Profile Scraper is a production-ready Apify actor that extracts structured data from public X (Twitter) profiles and searches. It lets you scrape tweets and optional profile metadata in clean JSON, making it a reliable Twitter profile data scraper and X profile scraper for marketers, developers, data analysts, and researchers. Use it to scrape Twitter profiles, run keyword-based searches, or export Twitter user profiles at scale for lead gen, competitor monitoring, and audience analysis.

What data / output can you get?

Below are the core fields this Twitter profile extractor saves to the Apify dataset. Outputs are individual tweet objects with optional profile data attached.

Data type	Description	Example value
id_str	Unique tweet ID (string)	"1519480761749016577"
full_text	Full tweet text	"Next I'm buying Coca-Cola to put the cocaine back in"
created_at	Tweet creation timestamp	"Thu Apr 28 00:56:58 +0000 2022"
favorite_count	Number of likes	4289223
retweet_count	Number of retweets	594435
reply_count	Number of replies	170050
quote_count	Number of quotes	167104
bookmark_count	Number of bookmarks (defaults to 0 if missing)	21112
conversation_id_str	Conversation/thread ID	"1519480761749016577"
user_id_str	Author user ID (string)	"44196397"
lang	Tweet language code	"en"
is_quote_status	Whether tweet is a quote	false
favorited	Whether favorited by the viewing context	false
retweeted	Whether retweeted by the viewing context	false
bookmarked	Whether bookmarked by the viewing context (defaults to false if missing)	false
display_text_range	Text display range indices	[0, 52]
entities	Tweet entities; always includes timestamps array	{"hashtags": [], "symbols": [], "user_mentions": [], "urls": [], "timestamps": []}
extended_entities	Extended media entities; may be null	null
user	Legacy profile object for the tweet author (included when addUserInfo is true)	{"screen_name": "elonmusk", "name": "Elon Musk", ...}

Notes:

• The user field contains the legacy profile object returned by X/Twitter (e.g., screen_name, name, description, followers_count, friends_count, statuses_count, profile_image_url, profile_banner_url, verified, created_at, etc.).
• When onlyUserInfo is enabled, outputs may be a single {"user": {...}} object per input without tweets.
• When enabled, users that are not found or suspended can be included with flags: {"user": {"screen_name": "
• You can export results from the Apify dataset to JSON, CSV, or Excel.

Key features

• 🔐 Resilient authorization header capture
  Automatically obtains the required authorization header using headless Chromium (Playwright), improving reliability for the Twitter user scraper workflow without manual tokens.

• 🧵 Tweet extraction with optional profile data
  Collect tweet-level fields (text, timestamps, engagement counters, entities) and include legacy user info per tweet when addUserInfo is enabled.

• 🧠 Keyword search or user timelines
  Works both as a Twitter profile scraper and a Twitter user profile scraper for timelines, and as a Twitter profile crawler for keyword-based searches.

• 📦 Bulk input & live dataset streaming
  Accepts multiple Twitter URLs, usernames, user IDs, or keywords via startUrls and streams items directly to the Apify dataset for large-scale runs.

• 🔄 Smart proxy fallback
  Built-in proxy logic escalates from no proxy → datacenter → residential with retries, reducing blocks for high-volume runs of this Twitter profile scraping tool.

• 🎯 Profile-only mode + status flags
  Use onlyUserInfo to export Twitter user profiles (legacy object) without tweets. Optionally include not-found or suspended accounts in the output.

• 🧰 Developer-friendly Python stack
  Built with Python, Playwright, aiohttp, and the Apify SDK. Ideal for integrations with internal pipelines or to download Twitter profile data through the Apify API.

• 📤 Flexible exports
  Export Twitter user profiles and tweets to JSON, CSV, or Excel from the Apify dataset to plug into BI tools, CRMs, or analysis scripts.

How to use Twitter Profile Scraper - step by step

1. Sign up or log in to Apify
   Access the actor from your Apify account to get started quickly.

2. Open the actor and add inputs
   In the Input tab, paste a list of items into startUrls. Accepted formats:

   • Profile URLs: https://x.com/elonmusk
   • Usernames: elonmusk
   • User IDs: 44196397
   • Keywords: tesla

3. Set tweet limits and ordering

   • maxTweets controls how many tweets to fetch per user or keyword.
   • sortOrder supports chronological or relevance (from the API).

4. Choose profile/tweet options

   • addUserInfo: include profile legacy data with each tweet.
   • onlyUserInfo: output only the user profile object (no tweets).
   • addNotFoundUsersToOutput / addSuspendedUsersToOutput: include status flags when applicable.

5. Configure proxy behavior (optional)
   Use proxyConfiguration to enable Apify Proxy. The actor can automatically fall back to datacenter and then residential proxies with retries if requests are blocked.

6. Run the actor
   Click Start. The Twitter account scraper fetches tweets and profile data, saving items live to the dataset.

7. Download results
   Go to the Dataset tab and export to JSON, CSV, or Excel for analysis, enrichment, or integration.

Pro Tip: To scrape Twitter profiles at scale and automate pipelines, schedule runs and pull results programmatically via the Apify API. You can also scrape Twitter profiles by keywords alongside usernames or IDs in the same run.

Use cases

Use case name	Description
Marketing – competitor content tracking	Monitor tweet frequency and engagement (likes, replies, quotes, bookmarks) from target accounts to inform content strategy.
Lead gen – audience & bios enrichment	Enrich records with legacy user fields (screen_name, name, description) and export Twitter user profiles for outreach workflows.
Research – trend and conversation analysis	Pull keyword-based timelines and structured entities for topic modeling, language analysis, or event studies.
Product – feature feedback mining	Aggregate product-related posts with conversation_id_str for thread analysis and faster issue triage.
Data engineering – API pipelines	Stream structured tweet JSON into data warehouses and ML pipelines using Apify storage and exports.
Journalism – public figure monitoring	Track official accounts at scale with chronological ordering to capture narratives over time.

Why choose Twitter Profile Scraper?

This Twitter profile extractor is engineered for precision, reliability, and automation on Apify’s infrastructure.

• ✅ Structured, reliable outputs: Consistent tweet fields with optional legacy user objects for clean downstream use.
• 🔄 Robust access strategy: Headless token capture plus automatic proxy fallback reduces blocks during peak loads.
• 📈 Built for scale: Works as a Twitter profile scraping tool and X user scraper with bulk inputs and live dataset streaming.
• 🧑‍💻 Developer-ready: Python + Playwright + Apify SDK stack is ideal for API-driven pipelines and integrations.
• 🛡️ Ethical by design: Targets publicly available data and provides controls for handling unavailable or suspended accounts.
• 💸 Cost- and time-efficient: Save engineering time with a production-ready Twitter user data extractor rather than unstable DIY scripts.

In short, it’s a dependable Twitter profile crawler alternative to fragile extensions or ad‑hoc scripts, built for teams that need repeatable, automated results.

Is it legal / ethical to use Twitter Profile Scraper?

Yes—when done responsibly. This actor is designed to extract publicly available data from X (Twitter). Users are responsible for ensuring compliance with Twitter’s Terms of Service and applicable data regulations.

Guidelines for compliant use:

• Only collect publicly available information.
• Avoid private data, login-only content, or any attempt to bypass security.
• Respect platform rate limits and robots/security controls.
• Align usage with data protection laws (e.g., GDPR, CCPA) and your organization’s policies.
• Consult your legal team for edge cases or sensitive use cases.

Input parameters & output format

Input JSON example

{
  "startUrls": [
    "https://x.com/elonmusk",
    "elonmusk",
    "44196397",
    "tesla"
  ],
  "maxTweets": 100,
  "sortOrder": "chronological",
  "addUserInfo": true,
  "onlyUserInfo": false,
  "addNotFoundUsersToOutput": false,
  "addSuspendedUsersToOutput": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Parameter reference:

• startUrls (array, required): List of Twitter profile URLs (e.g., https://x.com/username), usernames (e.g., username), user IDs (e.g., 44196397), or search keywords (e.g., tesla). Supports bulk input. Default: none.
• maxTweets (integer): Maximum number of tweets to fetch per user or keyword. Default is 10. Minimum: 1, Maximum: 100. Note: The actor enforces a minimum of 10 and caps at 1000 internally.
• sortOrder (string): Sort order for tweets (currently supports chronological order from API). Enum: ["chronological", "relevance"]. Default: "chronological".
• addUserInfo (boolean): Include user profile data (legacy) in each tweet output. Default: true.
• onlyUserInfo (boolean): If enabled, only fetch user profile information without tweets. Default: false.
• addNotFoundUsersToOutput (boolean): Include users that were not found in the output with notFound flag. Default: false.
• addSuspendedUsersToOutput (boolean): Include suspended users in the output with suspended flag. Default: false.
• proxyConfiguration (object): Proxy configuration. By default, no proxy is used. If Twitter blocks requests, the actor will automatically fallback to datacenter proxy, then residential proxy with retries. Prefill: {"useApifyProxy": false}.

Output JSON example (tweet item)

{
  "user": {
    "screen_name": "elonmusk",
    "name": "Elon Musk",
    "description": "Mars & Cars, Chips & Dips",
    "followers_count": 229033543,
    "friends_count": 1226,
    "statuses_count": 89153,
    "favourites_count": 182734,
    "listed_count": 165176,
    "created_at": "Tue Jun 02 20:12:29 +0000 2009",
    "verified": true,
    "profile_image_url": "https://pbs.twimg.com/profile_images/...",
    "profile_banner_url": "https://pbs.twimg.com/profile_banners/...",
    "default_profile": false,
    "default_profile_image": false,
    "entities": {
      "description": {
        "urls": []
      }
    }
  },
  "id_str": "1519480761749016577",
  "full_text": "Next I'm buying Coca-Cola to put the cocaine back in",
  "created_at": "Thu Apr 28 00:56:58 +0000 2022",
  "favorite_count": 4289223,
  "retweet_count": 594435,
  "reply_count": 170050,
  "quote_count": 167104,
  "bookmark_count": 21112,
  "conversation_id_str": "1519480761749016577",
  "user_id_str": "44196397",
  "lang": "en",
  "is_quote_status": false,
  "favorited": false,
  "retweeted": false,
  "bookmarked": false,
  "display_text_range": [0, 52],
  "entities": {
    "hashtags": [],
    "symbols": [],
    "user_mentions": [],
    "urls": [],
    "timestamps": []
  },
  "extended_entities": null
}

Additional output shapes:

• Profile-only (when onlyUserInfo is true): {"user": { ...legacy user object... }}
• Not found or suspended (when enabled): {"user": {"screen_name": "

Notes:

• entities.timestamps is always present (an empty array if no values are available).
• extended_entities may be null when no media metadata is present.
• If addUserInfo is false, the user object is omitted from tweet items.

FAQ

Do I need a login or API keys to use this Twitter user scraper?

No. The actor automatically captures the required authorization header using headless Chromium (Playwright). You don’t need to supply credentials to scrape public data.

Can I scrape by username, profile URL, user ID, or keyword?

Yes. Provide any mix of profile URLs, usernames, numeric user IDs, or search keywords inside startUrls. The actor will detect the type and run user timeline or keyword search accordingly.

Can this export Twitter user profiles only?

Yes. Enable onlyUserInfo to fetch and output only legacy user profile information without tweets.

Does it include profile data with each tweet?

Yes, when addUserInfo is true, each tweet item includes a user legacy object (e.g., screen_name, name, description, followers_count, etc.). Set addUserInfo to false to omit it.

How many tweets can I scrape per input?

You control this via maxTweets. The schema default is 10, and the actor enforces a minimum of 10 and caps at 1000 internally. Actual throughput may vary based on rate limits.

What happens if a user is suspended or not found?

If you enable addSuspendedUsersToOutput or addNotFoundUsersToOutput, the actor will include an item with the relevant flag (suspended or notFound) alongside the provided handle.

Is scraping Twitter/X legal?

Scraping public data is generally legal when done responsibly. Collect only public information, respect Twitter’s Terms of Service and rate limits, and comply with applicable data protection laws.

What formats can I download the results in?

You can export your dataset to JSON, CSV, or Excel directly from Apify, making it easy to integrate with BI tools, CRMs, or pipelines.

Closing CTA / Final thoughts

The Twitter Profile Scraper is built for structured, scalable extraction of public tweets and optional profile metadata from X (Twitter). It delivers clean fields, resilient access, and bulk automation for marketers, developers, analysts, and researchers. Use the Apify API to integrate this Twitter profile extractor into your data workflows, or schedule runs to keep datasets fresh. Start extracting smarter insights from Twitter profiles and searches—at scale.