Twitter Profile Scraper

Twitter Profile Scraper

Twitter Profile Scraper is a production-ready Apify actor that extracts public Twitter/X profile and tweet data at scale for marketers, developers, data analysts, and researchers. It solves the pain of manual copy-paste and API hurdles by collecting structured tweet metadata and legacy profile info directly from public pages — ideal for social listening, lead gen, and competitor tracking. This Twitter profile data extractor helps you export Twitter profile data for analytics pipelines, enabling bulk workflows and repeatable automations.

What data / output can you get?

Below are the exact fields this Twitter user scraper tool writes to the Apify dataset. Records are pushed live during the run and can be exported to JSON or CSV.

Data field	Description	Example value
id_str	Unique tweet ID (string)	"1519480761749016577"
full_text	Full text of the tweet	"Next I'm buying Coca-Cola to put the cocaine back in"
created_at	Tweet creation time (Twitter format)	"Thu Apr 28 00:56:58 +0000 2022"
favorite_count	Number of likes	4289223
retweet_count	Number of retweets	594435
reply_count	Number of replies	170050
quote_count	Number of quotes	167104
bookmark_count	Number of bookmarks (defaults to 0 if missing)	21112
user_id_str	Author’s user ID (string)	"44196397"
conversation_id_str	Root conversation ID (string)	"1519480761749016577"
lang	Tweet language code	"en"
is_quote_status	Whether the tweet is a quote	false
favorited	Whether the tweet is favorited by the viewing session	false
retweeted	Whether the tweet is retweeted by the viewing session	false
bookmarked	Whether the tweet is bookmarked by the viewing session (defaults to false if missing)	false
display_text_range	Visible text range indices	[0, 52]
entities	Entities object (hashtags, mentions, urls) with an added timestamps array	{ "hashtags": [], "user_mentions": [], "urls": [], "timestamps": [] }
extended_entities	Extended entities (e.g., media) when available	null
user	Legacy user profile object for the tweet author	{ "screen_name": "elonmusk", "name": "Elon Musk", "description": "Mars & Cars, Chips & Dips", "followers_count": 229033543, "friends_count": 1226, "statuses_count": 89153, "created_at": "Tue Jun 02 20:12:29 +0000 2009", "verified": true, "profile_image_url": "https://pbs.twimg.com/profile_images/...", "profile_banner_url": "https://pbs.twimg.com/profile_banners/..." }

Notes:

• When addUserInfo is false, the user field is removed.
• If onlyUserInfo is true, the actor outputs a single object per input with user (legacy profile) only.
• If a user cannot be found or is suspended and you enable the respective flags, the actor can output lightweight records like {"user": {"screen_name": "

Key features

• ⚡ Robust proxy fallback & rotation — Automatic progression from direct to datacenter to residential proxies with retries to reduce blocks and keep runs reliable.
• 🧠 Authorization capture without API keys — Dynamically captures the required Bearer authorization via Playwright; perfect for a Twitter profile scraper without API credentials.
• 📚 Profile + tweet metadata — Optionally include legacy profile data in each tweet (addUserInfo) or fetch profiles only (onlyUserInfo) for a focused Twitter bio scraper workflow.
• 🔎 Keyword search support — Treat non-URL/handle inputs as keywords and collect search timeline tweets at scale.
• 📥 Bulk inputs — Feed lists of profile URLs, @handles, user IDs, or keywords via startUrls for a true bulk Twitter profile scraper.
• 🎛️ Flexible limits — Control tweet collection with maxTweets; the actor enforces a minimum of 10 and internally caps at 1000.
• 💾 Clean, structured exports — Export Twitter profile data and tweets from the dataset to JSON or CSV for downstream analytics and BI.
• 🧩 Developer-friendly — Works seamlessly with the Apify platform and API, making it easy to integrate into Python pipelines and automation stacks.
• 🛡️ Production-grade reliability — Live saving to the dataset, graceful handling of rate limits, and resilient pagination for uninterrupted runs.

How to use Twitter Profile Scraper - step by step

1. Create or log in to your Apify account
   You’ll run the actor in the Apify cloud and store results in an Apify dataset.

2. Open the Twitter Profile Scraper actor
   Find “Twitter Profile Scraper” in the Apify Store and click Try for free.

3. Add input data in startUrls
   Paste Twitter/X profile URLs, usernames, user IDs, or keywords. Examples:

   • https://x.com/elonmusk
   • mrbeast
   • 44196397
   • tesla

4. Configure options

   • maxTweets to control how many tweets to fetch per user/keyword
   • sortOrder to set chronological or relevance (as available)
   • addUserInfo to include legacy user data with each tweet
   • onlyUserInfo to fetch profile info only (no tweets)
   • addNotFoundUsersToOutput / addSuspendedUsersToOutput to flag edge cases
   • proxyConfiguration to enable Apify Proxy and allow automatic fallback

5. Start the run
   Click Start. The actor will capture authorization, resolve users or keywords, and begin fetching tweets with pagination.

6. Monitor progress
   Watch logs for pagination steps and any proxy fallbacks. Data is pushed live to the default dataset as it’s collected.

7. Download results
   Go to the Dataset tab for your run and export to JSON or CSV. Use these outputs for research, lead gen, or analytics pipelines.

Pro tip: Use the Apify API to pull dataset items directly into your data stack, or schedule runs to refresh influencer and competitor datasets automatically.

Use cases

Use case	Description
Marketing analytics – influencer tracking	Monitor verified or high-reach accounts, collect tweet performance (likes, replies, quotes, bookmarks), and analyze content cadence.
Competitor research – content strategy	Aggregate tweets from competitor handles to study topics, engagement patterns, and campaign timing.
Lead generation – audience signals	Use bios and legacy profile fields to build segmented lists from public profiles relevant to your niche.
Academic research – discourse analysis	Collect tweets by keyword or handle for language, sentiment, or trend studies with consistent metadata.
Social listening – keyword monitoring	Track keywords and hashtags via search inputs and export structured timelines for dashboards.
Data engineering – API pipelines	Automate dataset refreshes and consume JSON/CSV outputs from the Apify API into warehouses or ML pipelines.

Why choose Twitter Profile Scraper?

Built for precision, automation, and reliability, this X profile scraper scales from single handles to large profile lists with structured outputs you can trust.

• 🎯 Accurate, structured results — Clean tweet fields and legacy profile objects ready for analysis without heavy post-processing.
• 🌍 Scalable batch operations — Feed many profiles/keywords via startUrls to run a true Twitter profile crawler at scale.
• 🧑‍💻 Developer access — Integrate via Apify’s API in Python scripts and CI/CD workflows for repeatable automations.
• 🔐 Safe & robust — Automatic proxy fallback and retries reduce blocks versus brittle browser extensions and unstable tools.
• 💸 Cost-optimised & reliable — Live dataset writes and resilient pagination reduce reruns and wasted compute.
• 🔌 Integration-ready — Export Twitter profile data to JSON/CSV and plug into CRMs, BI tools, or data lakes.

In short: a production-grade Twitter profile information scraper that outperforms quick hacks and extensions with stability and clean data.

Is it legal / ethical to use Twitter Profile Scraper?

Yes — when done responsibly. This actor extracts publicly available Twitter/X data and does not target private or login-only content.

Guidelines for compliant use:

• Collect only publicly visible data and respect the platform’s Terms of Service.
• Avoid scraping private profiles or attempting to bypass security.
• Use proxies ethically and rate-limit-aware practices to prevent abuse.
• Ensure your use aligns with data protection laws (e.g., GDPR/CCPA) and consult your legal counsel for edge cases.

Input parameters & output format

Example JSON input

{
  "startUrls": [
    "https://x.com/elonmusk",
    "mrbeast",
    "44196397",
    "tesla"
  ],
  "maxTweets": 100,
  "sortOrder": "chronological",
  "addUserInfo": true,
  "onlyUserInfo": false,
  "addNotFoundUsersToOutput": false,
  "addSuspendedUsersToOutput": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Parameter reference

Field	Type	Required	Default	Description
startUrls	array	Yes	—	List of Twitter profile URLs (e.g., https://x.com/username), usernames (e.g., username), user IDs (e.g., 44196397), or search keywords (e.g., tesla). Supports bulk input.
maxTweets	integer	No	10	Maximum number of tweets to fetch per user or keyword. (Schema range: 1–100; actor enforces a minimum of 10 and internally caps at 1000.)
sortOrder	string (enum)	No	chronological	Sort order for tweets (currently supports chronological order from API). Allowed values: “chronological”, “relevance”.
addUserInfo	boolean	No	true	Include user profile data (legacy) in each tweet output.
onlyUserInfo	boolean	No	false	If enabled, only fetch user profile information without tweets.
addNotFoundUsersToOutput	boolean	No	false	Include users that were not found in the output with notFound flag.
addSuspendedUsersToOutput	boolean	No	false	Include suspended users in the output with suspended flag.
proxyConfiguration	object	No	{"useApifyProxy": false}	Proxy configuration. By default, no proxy is used. If Twitter blocks requests, the actor will automatically fallback to datacenter proxy, then residential proxy with retries.

Example JSON output

{
  "user": {
    "screen_name": "elonmusk",
    "name": "Elon Musk",
    "description": "Mars & Cars, Chips & Dips",
    "followers_count": 229033543,
    "friends_count": 1226,
    "statuses_count": 89153,
    "favourites_count": 182734,
    "listed_count": 165176,
    "created_at": "Tue Jun 02 20:12:29 +0000 2009",
    "verified": true,
    "profile_image_url": "https://pbs.twimg.com/profile_images/...",
    "profile_banner_url": "https://pbs.twimg.com/profile_banners/...",
    "default_profile": false,
    "default_profile_image": false,
    "entities": {
      "description": {
        "urls": []
      }
    }
  },
  "id_str": "1519480761749016577",
  "full_text": "Next I'm buying Coca-Cola to put the cocaine back in",
  "created_at": "Thu Apr 28 00:56:58 +0000 2022",
  "favorite_count": 4289223,
  "retweet_count": 594435,
  "reply_count": 170050,
  "quote_count": 167104,
  "bookmark_count": 21112,
  "conversation_id_str": "1519480761749016577",
  "user_id_str": "44196397",
  "lang": "en",
  "is_quote_status": false,
  "favorited": false,
  "retweeted": false,
  "bookmarked": false,
  "display_text_range": [0, 52],
  "entities": {
    "hashtags": [],
    "symbols": [],
    "user_mentions": [],
    "urls": [],
    "timestamps": []
  },
  "extended_entities": null
}

Output variations:

• If addUserInfo is false, the user object is omitted from tweet records.
• If onlyUserInfo is true, the actor outputs one object per input with user only (no tweet fields).
• With addNotFoundUsersToOutput or addSuspendedUsersToOutput enabled, records may include notFound: true or suspended: true alongside a minimal user object containing screen_name.

FAQ

What does this Twitter Profile Scraper collect?

It collects tweet-level fields like id_str, full_text, created_at, engagement counts (likes, replies, retweets, quotes, bookmarks), language, entities, and more. When addUserInfo is enabled, it also includes the legacy user profile object with fields such as screen_name, name, description, followers_count, and profile images.

Can I scrape by handle, URL, user ID, or keyword?

Yes. Provide profile URLs, @handles, numeric user IDs, or keywords in startUrls. Non-URL/handle inputs that don’t look like IDs are treated as keyword searches.

Does it require API keys?

No. The actor dynamically captures the required authorization header using Playwright and then queries public endpoints — a practical approach for an X profile scraper without API keys.

How many tweets can I fetch per profile?

You control this with maxTweets. The UI schema allows 1–100; the actor enforces a minimum of 10 and internally caps at 1000 to balance performance and reliability.

Can I export the data?

Yes. You can export results from the Apify dataset to JSON or CSV and feed them into BI tools, CRMs, or data pipelines to export Twitter profile data efficiently.

Can I include only profile information without tweets?

Yes. Set onlyUserInfo to true to fetch the legacy profile object only. You can also include not-found or suspended markers using addNotFoundUsersToOutput and addSuspendedUsersToOutput.

How does it handle blocks and rate limits?

The actor features automatic proxy fallback: direct connection → datacenter proxy → residential proxy with retries. This reduces blocks and maintains stable scraping at scale.

Does it work with Python and automation tools?

Yes. Use the Apify API from Python to pull datasets or trigger runs. It’s easy to integrate this Twitter profile scraper Python workflow into larger automations.

Closing CTA / Final thoughts

Twitter Profile Scraper is built for fast, reliable extraction of public Twitter/X profile and tweet data at scale. With robust proxy fallback, flexible inputs (URLs, handles, IDs, keywords), and clean JSON/CSV outputs, it’s ideal for marketers, analysts, researchers, and developers. Use the Apify API to automate refreshes, plug results into your data stack, and keep tabs on influencers and competitors with confidence. Start extracting smarter social insights today.