Twitter Profile Scraper

Twitter Profile Scraper

The Twitter Profile Scraper is a fast, scalable Apify actor that extracts public Twitter/X data for marketers, developers, data analysts, and researchers. It turns profile timelines and keyword searches into structured records with tweet text, engagement metrics, language, and legacy profile info — perfect for workflows like a Twitter profile data extractor, Twitter user scraper, or Twitter profile lead generation tool at scale. Use it to scrape Twitter profiles, export structured datasets, and automate pipelines reliably.

What data / output can you get?

This Twitter profile scraping tool outputs structured tweet objects and, optionally, user legacy profile data. Below are representative fields (as they appear in the dataset) with concrete examples.

Data type	Description	Example value
user.screen_name	Legacy profile handle included when addUserInfo=true	"elonmusk"
user.name	Legacy display name	"Elon Musk"
user.description	Legacy profile bio	"Mars & Cars, Chips & Dips"
user.verified	Legacy verification status	true
user.profile_image_url	Legacy profile image URL	"https://pbs.twimg.com/profile_images/..."
id_str	Tweet ID (string)	"1519480761749016577"
full_text	Full tweet text	"Next I'm buying Coca-Cola to put the cocaine back in"
created_at	Tweet creation time	"Thu Apr 28 00:56:58 +0000 2022"
favorite_count	Likes on the tweet	4289223
retweet_count	Retweets on the tweet	594435
reply_count	Replies on the tweet	170050
quote_count	Quotes on the tweet	167104

Bonus fields include conversation_id_str, user_id_str, lang, is_quote_status, favorited, retweeted, bookmarked, bookmark_count, display_text_range, entities (with timestamps array), and extended_entities. You can export results from Apify in JSON, CSV, or Excel to feed CRMs, dashboards, or analytics pipelines for your Twitter account scraper use cases.

Key features

• ⚡ Automatic authorization & resilient fetching
  Captures a live Bearer token via Playwright and fetches tweets through the responsive web API for speed and stability.

• 🔁 Smart proxy fallback & retries
  Built-in proxy logic escalates from no proxy → datacenter → residential with exponential backoff on failures (ideal for a Public Twitter profile scraper or X profile scraper).

• 📚 Multi-input support (URLs, usernames, IDs, keywords)
  Works as a Twitter handle scraper and Twitter profile crawler. Provide profile URLs, @handles, numeric user IDs, or keyword queries — great for bulk lead gen and research.

• 🎛️ Flexible output controls

  • addUserInfo: include legacy user profile data in each tweet object
  • onlyUserInfo: fetch only profile info (no tweets)
  • addNotFoundUsersToOutput / addSuspendedUsersToOutput: include flags for unavailable accounts

• 📦 Live dataset writes & easy export
  Saves items as they’re scraped for reliability and quick access. Export Twitter profiles and tweets to JSON/CSV/Excel in one click.

• 👨‍💻 Developer-friendly automation
  Designed for API pipelines and Python/SDK integrations. Ideal for teams building Twitter user data extraction software or a Twitter profile scraping tool into ETL workflows.

• 🛡️ Production-ready infrastructure
  Headless Playwright + aiohttp + Apify platform for uptime, scaling, and consistent runs across large campaigns.

How to use Twitter Profile Scraper - step by step

1. Sign in to Apify
   Create a free account or log in to start your first run.

2. Open the Twitter Profile Scraper actor
   Find “Twitter Profile Scraper” in the Apify Store.

3. Add input data
   Paste items into startUrls as a string list. Accepts:

   • Profile URLs: https://x.com/username
   • Usernames: username (no @ needed)
   • User IDs: 44196397
   • Keywords: tesla

4. Configure settings

   • maxTweets: limit items per user/keyword (runtime enforces a minimum of 10 and a maximum of 1000)
   • sortOrder: chronological or relevance (API supports chronological)
   • addUserInfo / onlyUserInfo: include or focus on legacy profile info
   • addNotFoundUsersToOutput / addSuspendedUsersToOutput: include flags for unavailable users
   • proxyConfiguration: optionally enable Apify Proxy; actor auto-fallbacks to datacenter then residential if blocked

5. Start the run
   The actor will collect the authorization header automatically, resolve user IDs when needed, and begin fetching tweets or profiles.

6. Monitor progress
   Items are pushed live to the dataset as they’re collected.

7. Download results
   Export to JSON, CSV, or Excel from the dataset to integrate with analytics, CRMs, or BI tools.

Pro Tip: Provide a mix of profile URLs and keyword queries in one run to combine a Twitter profile data extractor with topic monitoring for research and competitor analysis.

Use cases

Use case name	Description
Marketing analytics + influencer tracking	Aggregate tweet performance and legacy profile data to compare engagement across creators.
Lead generation from public profiles	Filter by bios and activity to fuel campaigns with a Twitter profile lead generation tool.
Competitor monitoring for strategy	Track timelines of competitor accounts to analyze messaging and cadence.
Social listening via keyword search	Collect keyword-based tweets for topics, events, or brand terms without relying on paid APIs.
Academic & policy research	Build longitudinal datasets of public discourse for studies, experiments, or reproducibility.
Data engineering pipelines (API/SDK)	Automate export Twitter profiles and tweets into warehouses for enrichment and ML.
Public Twitter profile audits	Fetch only user legacy data (onlyUserInfo) to build structured profile catalogs.

Why choose Twitter Profile Scraper?

This actor is engineered for precision, automation, and reliability — a robust alternative to unstable extensions or manual scripts.

• ✅ Accurate, structured outputs ready for analysis
• ⚙️ Handles URLs, usernames, user IDs, and keywords in bulk
• 🔌 Developer access via the Apify platform for API/SDK integrations
• 🧰 Flexible toggles for including legacy user data or profile-only runs
• 🛡️ Safe fallback to Apify Proxy (datacenter → residential) with retries
• 💰 Cost-effective scaling on production-grade infrastructure

In short: a reliable Twitter user scraper that beats brittle browser add-ons and one-off scripts for teams that need repeatable, scalable results.

Is it legal / ethical to use Twitter Profile Scraper?

Yes — when used responsibly on publicly available data. This actor targets public Twitter/X endpoints and legacy profile information without logging in.

Guidelines for compliant use:

• Only collect publicly accessible data; avoid private or gated content.
• Respect Twitter/X Terms of Service and rate limits.
• Do not attempt to bypass security or scrape private profiles.
• Ensure your use complies with applicable data protection laws (e.g., GDPR/CCPA).
• Consult your legal team for edge cases or jurisdiction-specific requirements.

Input parameters & output format

Example JSON input

{
  "startUrls": [
    "https://x.com/elonmusk",
    "mrbeast",
    "44196397",
    "tesla"
  ],
  "maxTweets": 100,
  "sortOrder": "chronological",
  "addUserInfo": true,
  "onlyUserInfo": false,
  "addNotFoundUsersToOutput": false,
  "addSuspendedUsersToOutput": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Parameters

• startUrls (array, required): List of Twitter profile URLs (e.g., https://x.com/username), usernames (e.g., username), user IDs (e.g., 44196397), or search keywords (e.g., tesla). Supports bulk input. Default: none.
• maxTweets (integer, optional): Maximum number of tweets to fetch per user or keyword. Schema default is 10 (min 1, max 100). At runtime, the actor enforces a minimum of 10 and a maximum of 1000.
• sortOrder (string, optional): Sort order for tweets. Options: chronological, relevance. Default: chronological.
• addUserInfo (boolean, optional): Include user profile data (legacy) in each tweet output. Default: true.
• onlyUserInfo (boolean, optional): If enabled, only fetch user profile information without tweets. Default: false.
• addNotFoundUsersToOutput (boolean, optional): Include users that were not found in the output with notFound flag. Default: false.
• addSuspendedUsersToOutput (boolean, optional): Include suspended users in the output with suspended flag. Default: false.
• proxyConfiguration (object, optional): Proxy configuration. By default, no proxy is used. If Twitter blocks requests, the actor will automatically fallback to datacenter proxy, then residential proxy with retries. Default: {"useApifyProxy": false}.

Example JSON output (tweet with legacy user info)

{
  "user": {
    "screen_name": "elonmusk",
    "name": "Elon Musk",
    "description": "Mars & Cars, Chips & Dips",
    "followers_count": 229033543,
    "friends_count": 1226,
    "statuses_count": 89153,
    "favourites_count": 182734,
    "listed_count": 165176,
    "created_at": "Tue Jun 02 20:12:29 +0000 2009",
    "verified": true,
    "profile_image_url": "https://pbs.twimg.com/profile_images/...",
    "profile_banner_url": "https://pbs.twimg.com/profile_banners/...",
    "default_profile": false,
    "default_profile_image": false,
    "entities": {
      "description": {
        "urls": []
      }
    }
  },
  "id_str": "1519480761749016577",
  "full_text": "Next I'm buying Coca-Cola to put the cocaine back in",
  "created_at": "Thu Apr 28 00:56:58 +0000 2022",
  "favorite_count": 4289223,
  "retweet_count": 594435,
  "reply_count": 170050,
  "quote_count": 167104,
  "bookmark_count": 21112,
  "conversation_id_str": "1519480761749016577",
  "user_id_str": "44196397",
  "lang": "en",
  "is_quote_status": false,
  "favorited": false,
  "retweeted": false,
  "bookmarked": false,
  "display_text_range": [0, 52],
  "entities": {
    "hashtags": [],
    "symbols": [],
    "user_mentions": [],
    "urls": [],
    "timestamps": []
  },
  "extended_entities": null
}

Optional output variants (when toggles are used):

• onlyUserInfo=true (profile-only)

{
  "user": {
    "screen_name": "apify",
    "name": "Apify",
    "description": "The universal web scraping and automation platform.",
    "verified": true
  }
}

• addNotFoundUsersToOutput=true (user not found)

{
  "user": { "screen_name": "some_unknown_user" },
  "notFound": true
}

• addSuspendedUsersToOutput=true (user suspended)

{
  "user": { "screen_name": "suspended_example" },
  "suspended": true
}

Notes

• If addUserInfo=false, the user object is removed from tweet records.
• Entities always include a timestamps array (added by the actor) and may be empty when not present in the source.
• Keyword inputs produce tweet objects identical to profile timelines, deduplicated by id_str.

FAQ

Do I need to log in to use this Twitter Profile Scraper?

No. The actor uses headless browsing to obtain a public authorization header and then fetches public data. It does not require your Twitter/X credentials.

Can I scrape by username, profile URL, user ID, or keyword?

Yes. The startUrls array accepts profile URLs, plain usernames, numeric user IDs, and free-text keywords. This lets you run a Twitter profile crawler and topic monitor in one job.

How many tweets can I extract per input?

You control this with maxTweets. The schema default is 10 (min 1, max 100), and at runtime the actor enforces a minimum of 10 and a maximum of 1000 per user or keyword.

Can I include legacy profile data with each tweet?

Yes. Set addUserInfo to true to include the legacy user object per tweet. To fetch just profiles (no tweets), set onlyUserInfo to true.

What happens if a user is suspended or not found?

If you enable addNotFoundUsersToOutput and/or addSuspendedUsersToOutput, the actor will push a record with the appropriate notFound or suspended flag so you can track input coverage.

Does it support proxies and anti-blocking?

Yes. If requests are blocked, the actor automatically falls back from no proxy to Apify datacenter proxy and then to residential proxy with retries, improving success for a Public Twitter profile scraper at scale.

What formats can I export to?

Results are written to an Apify dataset, which you can download as JSON, CSV, or Excel. This is ideal for a Twitter profile data extractor feeding analytics, CRMs, or BI tools.

Can I integrate this with Python or other automation tools?

Yes. You can run the actor via the Apify API and connect it to Python scripts, n8n, Make, or internal ETL to build a Twitter user data extraction software pipeline.

Closing CTA / Final thoughts

Twitter Profile Scraper is built for accurate, scalable extraction of public Twitter/X timelines and legacy profile information. With flexible inputs, proxy fallback, and structured outputs, it’s ideal for marketers, analysts, researchers, and developers. Export Twitter profiles and tweets in JSON/CSV/Excel, or plug it into your API/Python automation pipeline. Start extracting smarter Twitter insights — reliably and at scale.