Twitter Profile Scraper

Twitter Profile Scraper

The Twitter Profile Scraper is a fast, Python-based Apify actor that extracts public data from X/Twitter profiles and searches. It turns profile URLs, usernames, user IDs, or keywords into structured tweet records with optional legacy user info — perfect for marketers, developers, data analysts, and researchers. As a Twitter profile data scraper and X profile scraper, it helps you build scalable pipelines for social listening, lead enrichment, and brand monitoring — at scale and without login.

What data / output can you get?

Below are the exact JSON fields this Twitter user scraper tool pushes to the Apify dataset. Each record is a tweet (from a profile or keyword search) with optional legacy user details.

Data type	Description	Example value
id_str	Tweet ID (string)	"1519480761749016577"
full_text	Full tweet text	"Next I'm buying Coca-Cola to put the cocaine back in"
created_at	Tweet creation time (string)	"Thu Apr 28 00:56:58 +0000 2022"
favorite_count	Likes count	4289223
retweet_count	Retweets count	594435
reply_count	Replies count	170050
quote_count	Quotes count	167104
bookmark_count	Bookmarks count (defaults to 0 if missing)	21112
user_id_str	Author user ID (string)	"44196397"
conversation_id_str	Conversation/thread ID	"1519480761749016577"
entities	Entities from tweet text, with timestamps array added	{"hashtags": [], "user_mentions": [], "urls": [], "timestamps": []}
extended_entities	Media entities if present; null otherwise	null
lang	Language code	"en"
is_quote_status	Whether the tweet is a quote	false
favorited	Whether the authenticating user favorited (legacy field)	false
retweeted	Whether the authenticating user retweeted (legacy field)	false
bookmarked	Whether the tweet was bookmarked (defaults to false if missing)	false
display_text_range	Visible text range indexes	[0, 52]
user	Legacy user profile object for the author (included unless disabled)	{"screen_name": "elonmusk", "name": "Elon Musk", ...}

Notes:

• Bonus variants: When onlyUserInfo is enabled, the actor outputs a single {"user": {...}} legacy profile object per input (no tweets). If a user is not found or suspended and the respective include flags are enabled, the actor outputs {"user": {"screen_name": ""}, "notFound": true} or {"user": {"screen_name": ""}, "suspended": true}.
• Export formats: Download your dataset as JSON or CSV from Apify. It’s a clean Twitter profile data extractor for analytics-ready pipelines.

Key features

• ⚡ Robust proxy fallback The actor automatically falls back from no proxy → datacenter → residential (with retries) when it encounters blocks (e.g., 401/403/429). This improves resilience for bulk Twitter profile scraper workflows.

• 🔍 Dual-mode input: profiles & keywords Feed it profile URLs, @usernames, user IDs, or raw keywords. It detects keywords to run an on-site search and collects matching tweets — ideal for a Twitter profile crawler and Twitter profile data extractor in one.

• 🧩 Optional legacy user info Include or exclude the legacy user object on each tweet with addUserInfo. You can also fetch only user profiles (onlyUserInfo) without any tweets — a simple Twitter bio scraper mode for enrichment.

• 🧪 Not-found & suspended flags If enabled, the actor can emit lightweight records for not-found or suspended inputs, helping you keep complete audit trails in bulk Twitter username scraper runs.

• 🐍 Developer friendly (Python-based) Built with Python, Playwright, and aiohttp for speed and reliability. Works great as a Twitter profile scraper API backend or integrated into your Twitter profile scraper Python pipeline.

• 💾 Live dataset writes Results are pushed as they’re scraped (live saving), so you can stream data into analytics workflows. Clean JSON/CSV output supports downstream automation.

• 🛡️ TOS-aware public scraping Designed for public data access and responsible operation. Combined with proxy options, it’s a stable X user profile scraper for production use.

How to use Twitter Profile Scraper - step by step

1. Sign up or log in to Apify Create a free account to get started with this X profile scraper.

2. Open the actor Find “Twitter Profile Scraper” in the Apify Store and click Try for free.

3. Add input data Paste your list into startUrls. You can mix:

   • Profile URLs: https://x.com/elonmusk
   • Usernames: mrbeast
   • User IDs: 44196397
   • Keywords: tesla

4. Configure options

   • maxTweets: Set how many tweets to fetch per input.
   • sortOrder: Choose "chronological" or "relevance" (API-dependent).
   • addUserInfo / onlyUserInfo: Include legacy user data or fetch only profiles.
   • addNotFoundUsersToOutput / addSuspendedUsersToOutput: Emit flags for missing/suspended inputs.
   • proxyConfiguration: Leave off by default, or use Apify Proxy for higher stability.

5. Start the run The Twitter user scraper tool will fetch tweets and optionally user info. It handles pagination and proxy fallback automatically.

6. Monitor progress Watch logs and the dataset for live results. Each item is pushed as soon as it’s parsed.

7. Export results Download your dataset as JSON or CSV. Use it in BI tools, CRMs, or scripts.

Pro tip: Combine keyword inputs with onlyUserInfo=false and addUserInfo=true to build rich, analytics-ready feeds that include tweet engagement and author profile context.

Use cases

Use case name	Description
Marketing analytics – influencer tracking	Aggregate tweets and legacy author info to analyze engagement patterns and audience signals for influencer shortlists.
Competitive intelligence – content monitoring	Track public posts from competitor handles and keyword themes to benchmark messaging, cadence, and traction.
Lead enrichment – profile context	Use onlyUserInfo to capture legacy profile details for accounts of interest and attach them to CRM records.
Brand monitoring – keyword feeds	Feed brand or product keywords to build near-real-time public tweet streams for social listening dashboards.
Research & academia – topic mining	Collect keyword-matched tweets and metadata for NLP experiments, trend analysis, and longitudinal studies.
Developer pipelines – API-ready JSON	Treat the dataset as a Twitter profile scraper API output for downstream ETL, alerting, or ML feature generation.

Why choose Twitter Profile Scraper?

A production-ready X user profile scraper that prioritizes precision, resilience, and control for public data collection.

• 🎯 Accurate, structured output: Clean JSON fields with optional legacy user blocks make downstream parsing easy.
• ⚡ Resilient at scale: Built-in proxy fallback (none → datacenter → residential, with retries) helps minimize blocks in bulk Twitter profile crawler runs.
• 🧩 Flexible inputs: Accepts profile URLs, @usernames, user IDs, and keywords in one list for streamlined batch jobs.
• 🐍 Developer-first: Python, Playwright, and aiohttp under the hood — ideal for scripting and pipeline integration.
• 🛡️ Responsible & public: Designed for public-data use; no login required.
• 💸 Predictable operations: Structured outputs and live saving reduce reruns and cleanup overhead versus unstable extensions.

Bottom line: It’s a reliable Twitter public profile scraper alternative to brittle browser plugins and ad‑hoc scripts.

Is it legal / ethical to use Twitter Profile Scraper?

Yes — when done responsibly. This actor targets public Twitter/X data and does not access private or login-only content.

Guidelines to stay compliant:

• Only collect publicly available information from profiles and searches.
• Respect Twitter/X Terms of Service and avoid bypassing security measures.
• Do not scrape private profiles or authenticated data.
• Observe rate limits and use proxies responsibly to reduce strain.
• Consult your legal team regarding jurisdiction-specific regulations (e.g., GDPR, CCPA) and your specific use case.

Input parameters & output format

Example JSON input

{
  "startUrls": [
    "https://x.com/elonmusk",
    "mrbeast",
    "44196397",
    "tesla"
    ],
  "maxTweets": 25,
  "sortOrder": "chronological",
  "addUserInfo": true,
  "onlyUserInfo": false,
  "addNotFoundUsersToOutput": false,
  "addSuspendedUsersToOutput": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Parameters

• startUrls (array, required): List of Twitter profile URLs (e.g., https://x.com/username), usernames (e.g., username), user IDs (e.g., 44196397), or search keywords (e.g., tesla). Supports bulk input. Default: none.
• maxTweets (integer): Maximum number of tweets to fetch per user or keyword. Minimum: 1, maximum: 100. Default: 10.
• sortOrder (string): Sort order for tweets (currently supports chronological order from API). One of: "chronological", "relevance". Default: "chronological".
• addUserInfo (boolean): Include user profile data (legacy) in each tweet output. Default: true.
• onlyUserInfo (boolean): If enabled, only fetch user profile information without tweets. Default: false.
• addNotFoundUsersToOutput (boolean): Include users that were not found in the output with notFound flag. Default: false.
• addSuspendedUsersToOutput (boolean): Include suspended users in the output with suspended flag. Default: false.
• proxyConfiguration (object): Proxy configuration. By default, no proxy is used. If Twitter blocks requests, the actor will automatically fallback to datacenter proxy, then residential proxy with retries. Default: none (no proxy).

Example JSON output (tweet with legacy user)

{
  "user": {
    "screen_name": "elonmusk",
    "name": "Elon Musk",
    "description": "Mars & Cars, Chips & Dips",
    "followers_count": 229033543,
    "friends_count": 1226,
    "statuses_count": 89153,
    "favourites_count": 182734,
    "listed_count": 165176,
    "created_at": "Tue Jun 02 20:12:29 +0000 2009",
    "verified": true,
    "profile_image_url": "https://pbs.twimg.com/profile_images/.../normal.jpg",
    "profile_banner_url": "https://pbs.twimg.com/profile_banners/.../header.jpg",
    "default_profile": false,
    "default_profile_image": false,
    "entities": {
      "description": {
        "urls": []
      }
    }
  },
  "id_str": "1519480761749016577",
  "full_text": "Next I'm buying Coca-Cola to put the cocaine back in",
  "created_at": "Thu Apr 28 00:56:58 +0000 2022",
  "favorite_count": 4289223,
  "retweet_count": 594435,
  "reply_count": 170050,
  "quote_count": 167104,
  "bookmark_count": 21112,
  "user_id_str": "44196397",
  "conversation_id_str": "1519480761749016577",
  "entities": {
    "hashtags": [],
    "symbols": [],
    "user_mentions": [],
    "urls": [],
    "timestamps": []
  },
  "extended_entities": null,
  "lang": "en",
  "is_quote_status": false,
  "favorited": false,
  "retweeted": false,
  "bookmarked": false,
  "display_text_range": [0, 52]
}

Other possible outputs when flags are used:

• Example: onlyUserInfo enabled (user profile only)

{
  "user": {
    "screen_name": "apify",
    "name": "Apify",
    "description": "Web scraping and automation platform.",
    "created_at": "Mon Jan 01 00:00:00 +0000 2018"
  }
}

• Example: not found or suspended (when include flags are enabled)

{ "user": { "screen_name": "unknown_handle" }, "notFound": true }

{ "user": { "screen_name": "suspended_handle" }, "suspended": true }

Notes:

• bookmark_count defaults to 0 and bookmarked to false if missing in the source.
• extended_entities may be null if the tweet has no media.
• entities.timestamps is always present as an array (added by the actor).

FAQ

Is there a free trial?

Yes. This actor offers 120 trial minutes so you can test the Twitter profile scraper API workflow before subscribing.

Do I need to log in to use it?

No. The actor collects public data and obtains necessary authorization headers dynamically without user login.

Can I scrape by keyword as well as by username?

Yes. Provide keywords in startUrls and the actor will run X search to collect matching tweets, making it a flexible Twitter profile crawler and public data extractor.

What output fields are included?

Each tweet record includes id_str, full_text, created_at, favorite_count, retweet_count, reply_count, quote_count, bookmark_count, user_id_str, conversation_id_str, entities, extended_entities, lang, is_quote_status, favorited, retweeted, bookmarked, display_text_range, and an optional legacy user object under user.

How many tweets can I fetch per input?

You can configure maxTweets per user or keyword. The input schema supports values between 1 and 100, with a default of 10.

Can it return only profile data without tweets?

Yes. Set onlyUserInfo to true to fetch just the legacy user profile data for each input — ideal for Twitter bio scraper and enrichment tasks.

How does it handle blocks or rate limits?

The actor features a built-in proxy fallback: it can move from no proxy to datacenter proxy and then to residential proxy with retries on specific HTTP statuses (401/403/429).

In what formats can I download the results?

You can export from the Apify dataset in JSON or CSV, making it suitable for analytics and pipeline ingestion.

Is it legal to use this scraper?

Yes, when used responsibly. It targets public data and is designed to be TOS-aware. Avoid private or authenticated content and ensure your use complies with applicable laws and platform policies.

Closing CTA / Final thoughts

Twitter Profile Scraper is built for fast, reliable extraction of public X/Twitter data from profiles and keyword searches. It delivers structured JSON/CSV with optional legacy user details, proxy fallback for resilience, and flexible inputs for bulk runs. Ideal for marketers, developers, data analysts, and researchers, it fits cleanly into Python-based and API-driven workflows. Start extracting smarter with a production-ready X user profile scraper that scales with your needs.