Twitter Profile Scraper

Twitter Profile Scraper

Twitter Profile Scraper is an Apify actor that extracts structured data from public X/Twitter profiles and searches — including tweet text, engagement metrics, language, and legacy user profile details — in ready-to-use formats. It solves the challenge of gathering reliable Twitter profile data at scale by combining headless browser automation with X’s public endpoints. Built for marketers, developers, data analysts, and researchers, this Twitter profile scraper tool works as an X profile scraper, Twitter account scraper, and Twitter user profile extractor — enabling automation and bulk harvesting for influencer research, lead generation, and brand monitoring at scale. 🚀

What data / output can you get?

Below are real fields the actor saves to the Apify dataset. Values come from tweet “legacy” objects and the embedded “user” legacy profile. Results can be exported as JSON, CSV, or Excel.

Data type	Description	Example value
id_str	Unique tweet ID (string)	"1519480761749016577"
full_text	Full tweet text	"Next I'm buying Coca-Cola to put the cocaine back in"
created_at	Tweet creation timestamp	"Thu Apr 28 00:56:58 +0000 2022"
favorite_count	Likes count	4289223
retweet_count	Retweets/Reposts count	594435
reply_count	Replies count	170050
quote_count	Quotes count	167104
bookmark_count	Bookmarks count (defaults to 0 if missing)	21112
conversation_id_str	Conversation/thread ID	"1519480761749016577"
user_id_str	Author user ID (string)	"44196397"
lang	Tweet language code	"en"
is_quote_status	Whether the tweet is a quote	false
favorited	Favorited by the session	false
retweeted	Retweeted by the session	false
bookmarked	Bookmarked by the session (defaults to false if missing)	false
display_text_range	Text display range indices	[0, 52]
entities	Tweet entities (hashtags, mentions, urls) with timestamps array added	{"hashtags": [], "urls": [], "timestamps": []}
extended_entities	Extended media entities (if present)	null
user.screen_name	Author handle (from legacy user)	"elonmusk"
user.name	Author display name	"Elon Musk"
user.description	Author bio	"Mars & Cars, Chips & Dips"
user.followers_count	Followers count	229033543
user.created_at	Account creation date	"Tue Jun 02 20:12:29 +0000 2009"
user.verified	Verification status	true
user.profile_image_url	Avatar URL	"https://pbs.twimg.com/profile_images/..."

Notes:

• The embedded user object can include additional legacy fields such as friends_count, statuses_count, favourites_count, listed_count, profile_banner_url, default_profile, default_profile_image, and entities.description.urls when available.
• When onlyUserInfo is enabled, the output contains just the legacy user object.
• If a user isn’t found or is suspended and the corresponding flags are enabled, the output includes notFound or suspended fields for that handle/ID.
• entities.timestamps is always present (added as an empty array if not provided by the source).

Key features

• ⚡ Robust proxy fallback & retries
  Automatically escalates from no proxy → datacenter → residential with exponential backoff to reduce blocks and keep runs reliable at scale.

• 🔍 Multi-input Twitter profile crawler
  Accepts X/Twitter profile URLs, usernames, user IDs, or keywords to scrape Twitter user profiles and timelines in flexible workflows.

• 📦 Clean, structured analytics outputs
  Captures tweet legacy fields plus a legacy user object per item, ideal for enrichment in a Twitter profile data scraper pipeline.

• 🧩 Developer-friendly and API-ready
  Seamlessly integrates with the Apify API for automation — perfect for Twitter profile scraper API use or Twitter profile scraper Python integrations.

• 📈 Batch scraping & bulk automation
  Feed a list of startUrls to run a bulk Twitter profile scraper for influencer lists, competitors, or cohorts.

• 🎚️ Flexible modes and controls
  Toggle addUserInfo to include/exclude embedded profiles or enable onlyUserInfo to fetch just profile metadata without tweets.

• 🔄 Live saving & resilience
  Pushes items incrementally and can include notFound/suspended flags for better pipeline observability.

• 🛡️ Production-grade X profile scraper
  Uses headless Chromium via Playwright to obtain authorization headers, then calls public endpoints — a reliable Twitter profile scraping software approach without brittle extensions.

How to use Twitter Profile Scraper - step by step

1. Create or log in to your Apify account
   Open the actor in the Apify Store.

2. Open the Twitter Profile Scraper actor
   Click Try for free to start a new run.

3. Add your input data
   Paste values into startUrls. Accepted formats include:

   • Profile URLs: https://x.com/elonmusk
   • Usernames: mrbeast
   • User IDs: 44196397
   • Keywords: tesla

4. Configure scraping options

   • maxTweets: Set the maximum tweets per user/keyword (default 10).
   • sortOrder: chronological or relevance.
   • addUserInfo: Include legacy user profile in each tweet item.
   • onlyUserInfo: Fetch only the user profile (no tweets).
   • addNotFoundUsersToOutput / addSuspendedUsersToOutput: Include flags for unavailable profiles.
   • proxyConfiguration: Optionally enable Apify Proxy to improve reliability.

5. Start the run
   The actor will obtain authorization, handle proxy fallback automatically, and begin fetching profile timelines and/or search results.

6. Monitor progress
   Items are saved live to the default dataset as they are collected.

7. Export your data
   Download results as JSON, CSV, or Excel from the Dataset tab, or access them via the Apify API for downstream pipelines.

Pro Tip: Use onlyUserInfo as a fast Twitter handle scraper for quick bio and metadata snapshots; switch to keyword inputs to run a Twitter profile crawler workflow over search timelines.

Use cases

Use case name	Description
Marketing + Influencer vetting	Aggregate tweets and legacy profile metrics to prioritize creators for campaigns and measure audience signals.
Competitor monitoring	Track content output and engagement toplines from competitor accounts with a repeatable Twitter profile scraping software workflow.
Lead generation + Enrichment	Enrich lead lists with bios, verification status, and profile metadata using a Twitter handle scraper pipeline.
Social listening + Trend analysis	Pull keyword-based timelines to analyze topics and sentiment across public conversations.
Academic + Policy research	Collect longitudinal tweet samples and profile metadata for reproducible studies using an X profile scraper.
Data engineering + API pipelines	Feed JSON into warehouses and ML workflows as a Twitter profile data scraper running on Apify.
Brand monitoring	Watch verified or high-impact accounts and export structured updates to dashboards.

Why choose Twitter Profile Scraper?

A production-ready Twitter profile data scraper built for precision, automation, and scale.

• 🎯 Accurate, structured outputs
  Emits normalized tweet legacy fields and a consistent legacy user object for analysis-ready datasets.

• 🚀 Scalable bulk scraping
  Handles lists of usernames/URLs/IDs/keywords for large campaigns and research cohorts.

• 💻 Developer access & integrations
  API-ready outputs for Python scripts, ETL jobs, and automation — a dependable Twitter profile scraper API.

• 🛡️ Safer, resilient operations
  Automatic proxy fallback and retries reduce blocks compared to brittle extensions or ad-hoc tools.

• 💰 Cost- and time-efficient
  Streamlined runs and clean data reduce post-processing overhead.

• 🧲 Flexible modes
  Toggle addUserInfo or onlyUserInfo to tailor payloads for enrichment or audits.

In short: a reliable Twitter profile crawler alternative to unstable tools — engineered for repeatable, high-quality data pipelines.

Is it legal / ethical to use Twitter Profile Scraper?

Yes — when used responsibly with public data. This actor targets publicly available X/Twitter content and does not access private profiles or authenticated data.

Guidelines for compliant use:

• Scrape only publicly available information.
• Respect X/Twitter Terms of Service and rate limits.
• Do not attempt to bypass security or access private data.
• Ensure your use aligns with applicable data protection laws (e.g., GDPR, CCPA).
• Consult your legal team for edge cases or jurisdiction-specific requirements.

Input parameters & output format

Example JSON input

{
  "startUrls": [
    "https://x.com/elonmusk",
    "mrbeast",
    "44196397",
    "tesla"
  ],
  "maxTweets": 100,
  "sortOrder": "chronological",
  "addUserInfo": true,
  "onlyUserInfo": false,
  "addNotFoundUsersToOutput": false,
  "addSuspendedUsersToOutput": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Parameters

Field	Type	Required	Default	Description
startUrls	array	Yes	None	List of Twitter profile URLs (e.g., https://x.com/username), usernames (e.g., username), user IDs (e.g., 44196397), or search keywords (e.g., tesla). Supports bulk input.
maxTweets	integer	No	10	Maximum number of tweets to fetch per user or keyword.
sortOrder	string	No	"chronological"	Sort order for tweets (currently supports chronological order from API). Allowed: "chronological", "relevance".
addUserInfo	boolean	No	true	Include user profile data (legacy) in each tweet output.
onlyUserInfo	boolean	No	false	If enabled, only fetch user profile information without tweets.
addNotFoundUsersToOutput	boolean	No	false	Include users that were not found in the output with notFound flag.
addSuspendedUsersToOutput	boolean	No	false	Include suspended users in the output with suspended flag.
proxyConfiguration	object	No	Not set (no proxy)	Proxy configuration. By default, no proxy is used. If Twitter blocks requests, the actor will automatically fallback to datacenter proxy, then residential proxy with retries.

Example JSON output (tweets with embedded user legacy)

{
  "user": {
    "screen_name": "elonmusk",
    "name": "Elon Musk",
    "description": "Mars & Cars, Chips & Dips",
    "followers_count": 229033543,
    "friends_count": 1226,
    "statuses_count": 89153,
    "favourites_count": 182734,
    "listed_count": 165176,
    "created_at": "Tue Jun 02 20:12:29 +0000 2009",
    "verified": true,
    "profile_image_url": "https://pbs.twimg.com/profile_images/...",
    "profile_banner_url": "https://pbs.twimg.com/profile_banners/...",
    "default_profile": false,
    "default_profile_image": false,
    "entities": {
      "description": {
        "urls": []
      }
    }
  },
  "id_str": "1519480761749016577",
  "full_text": "Next I'm buying Coca-Cola to put the cocaine back in",
  "created_at": "Thu Apr 28 00:56:58 +0000 2022",
  "favorite_count": 4289223,
  "retweet_count": 594435,
  "reply_count": 170050,
  "quote_count": 167104,
  "bookmark_count": 21112,
  "conversation_id_str": "1519480761749016577",
  "user_id_str": "44196397",
  "lang": "en",
  "is_quote_status": false,
  "favorited": false,
  "retweeted": false,
  "bookmarked": false,
  "display_text_range": [0, 52],
  "entities": {
    "hashtags": [],
    "symbols": [],
    "user_mentions": [],
    "urls": [],
    "timestamps": []
  },
  "extended_entities": null
}

Other possible outputs:

• Profile-only (when onlyUserInfo is true):

{
  "user": {
    "screen_name": "apify",
    "name": "Apify",
    "description": "Web scraping and automation platform",
    "created_at": "Mon Jan 01 00:00:00 +0000 2018",
    "verified": true
  }
}

• Not found or suspended (if enabled via addNotFoundUsersToOutput / addSuspendedUsersToOutput):

{
  "user": { "screen_name": "some_unknown_handle" },
  "notFound": true
}

{
  "user": { "screen_name": "possible_suspended" },
  "suspended": true
}

Notes:

• When addUserInfo is false, tweet items will omit the user field.
• entities.timestamps is always present (added as an empty array when not provided by the source).
• Some fields may be null if not present in the source objects.

FAQ

Is there a free tier or trial?

Yes. You can run the actor on Apify with trial minutes, then upgrade for larger workloads. This makes it practical to test as a Twitter profile scraper API before scaling.

Do I need to log in or provide cookies?

No. The actor collects public data and obtains the necessary authorization headers programmatically via Playwright, working as an X profile scraper without manual login.

Can I scrape by username, URL, user ID, or keyword?

Yes. startUrls accepts profile URLs, usernames, user IDs, and keywords. This flexibility lets you scrape Twitter user profiles and run keyword timelines with the same tool.

What data does the output include?

Each tweet includes id_str, full_text, created_at, engagement metrics, language, entities, and more. When addUserInfo is true, each item embeds the legacy user profile (e.g., screen_name, name, description, followers_count, verified).

Can I fetch only the user profile without tweets?

Yes. Enable onlyUserInfo to output just the legacy user object for each input handle/ID.

How does it handle unavailable or suspended accounts?

If addNotFoundUsersToOutput and/or addSuspendedUsersToOutput are enabled, the actor includes a minimal record with notFound or suspended flags.

Can I integrate this with Python and automation tools?

Yes. Access datasets via the Apify API and connect the Twitter profile scraping automation to Python scripts, Make, n8n, or your data pipeline.

How many tweets can I extract per input?

Control this with maxTweets (default 10). The actor paginates and saves items incrementally.

Is using this legal?

Scraping public X/Twitter data is generally permitted when you respect the platform’s Terms of Service and applicable laws. Avoid private data and excessive or abusive behavior.

Closing CTA / Final thoughts

Twitter Profile Scraper is built for clean, structured extraction of public X/Twitter timelines and legacy user profiles at scale. With flexible inputs, resilient proxy fallback, and API-ready outputs, it serves marketers, developers, analysts, and researchers alike. Use it as a Twitter profile data scraper for influencer analysis, a Twitter handle scraper for enrichment, or a keyword timeline harvester in your data pipelines — then export to JSON/CSV/Excel or connect via API to automate end-to-end. Start extracting smarter with a reliable Twitter profile crawler engineered for production workflows.