Twitter Profile Scraper

Twitter Profile Scraper

Twitter Profile Scraper is an Apify actor that extracts structured tweet data and profile metadata from public X/Twitter without manual effort. It solves the challenge of turning scattered profile timelines into clean, analysis-ready records, working as both an X profile scraper and a Twitter user scraper for marketers, developers, data analysts, and researchers. Use it to build repeatable, large-scale pipelines that scrape Twitter profiles and keyword timelines for analytics, enrichment, and reporting.

What data / output can you get?

The actor pushes structured JSON records to the Apify dataset. Each record represents a tweet with optional user metadata (legacy profile fields) attached. Below are core fields as they appear in the output:

Data type	Description	Example value
id_str	Unique tweet ID (string)	"1519480761749016577"
full_text	Tweet text (full)	"Next I'm buying Coca-Cola to put the cocaine back in"
created_at	Tweet creation time (string)	"Thu Apr 28 00:56:58 +0000 2022"
favorite_count	Likes count	4289223
retweet_count	Retweets/Reposts count	594435
reply_count	Replies count	170050
quote_count	Quote count	167104
bookmark_count	Bookmarks count (defaults to 0 if missing)	21112
conversation_id_str	Conversation/thread ID	"1519480761749016577"
user_id_str	Author user ID (string)	"44196397"
lang	Language code	"en"
is_quote_status	Whether tweet is a quote	false
favorited	Whether favorited by the viewing user	false
retweeted	Whether retweeted by the viewing user	false
bookmarked	Whether bookmarked by the viewing user (defaults to false)	false
display_text_range	Text display index range	[0, 52]
entities	Entities object (hashtags, mentions, urls) + timestamps array	{"hashtags": [], "user_mentions": [], "urls": [], "timestamps": []}
extended_entities	Extended media entities (if present)	null
user.screen_name	Author handle (legacy user object when included)	"elonmusk"
user.name	Author display name (legacy user object when included)	"Elon Musk"
user.description	Author bio (legacy user object when included)	"Mars & Cars, Chips & Dips"
user.followers_count	Followers count (legacy user object when included)	229033543

Notes:

• When addUserInfo is true (default), each tweet includes a legacy user object under user with fields such as screen_name, name, description, followers_count, friends_count, statuses_count, favourites_count, listed_count, created_at, verified, profile_image_url, profile_banner_url, default_profile, default_profile_image, entities, etc.
• In profile-only mode (onlyUserInfo = true), outputs contain only a single object with user: { ...legacy profile fields... } per input.
• If a user is not found or suspended and you enable the corresponding flags, the actor emits minimal objects with notFound: true or suspended: true.
• Export your dataset to JSON, CSV, or Excel directly from Apify.

Key features

• 🔎 Powerful timeline extraction
  Collect tweet objects with full_text, timestamps, entities, and engagement counts. Works as a fast Twitter profile data extractor and Twitter profile info scraper for public timelines.

• 🧩 User metadata toggle
  Include or exclude the legacy user object per tweet via addUserInfo for slimmer payloads or richer context.

• 🧱 Profile-only mode
  Enable onlyUserInfo to fetch just the profile’s legacy metadata (no tweets) — perfect for enrichment and quick profile audits with a Twitter profile metadata scraper approach.

• 🧠 Keyword search mode
  Provide keywords in startUrls to scrape public search timelines — useful when you need a Twitter profile crawler alternative that also supports topical discovery.

• 🔄 Robust proxy fallback
  Automatic escalation: none → datacenter → residential with retries. The actor dynamically captures authorization headers with Playwright and rotates proxies when blocked.

• 📥 Bulk input & live dataset writes
  Feed URLs, @handles, user IDs, or keywords to process many targets in one run. Items are saved as they’re scraped for reliability and scale.

• 🧪 Developer-friendly & API-ready
  Built as an Apify actor, it integrates with your stack as a Twitter profile scraping API. Trigger via API, connect to pipelines, or use from Python SDKs.

• 🧭 Sort parameter included
  Control sortOrder (chronological | relevance) to align with your workflow. The actor uses the platform’s chronological order support where applicable.

• 🧰 Production-focused reliability
  Playwright-powered collection of auth headers, resilient pagination, and clear notFound/suspended signaling across results.

How to use Twitter Profile Scraper - step by step

1. Sign up and log in to Apify
   Create a free account and access the Apify Console.

2. Find the actor
   Search for “Twitter Profile Scraper” in the Apify Store and open the actor page.

3. Add input data
   In startUrls, enter a list of:

   • Profile URLs: https://x.com/elonmusk
   • Usernames: elonmusk
   • User IDs: 44196397
   • Keywords: tesla (keyword mode scrapes public search timelines)

4. Configure settings

   • maxTweets: Limit tweets per user/keyword (default 10; UI range 1–100). Internally, the actor enforces a minimum of 10 and a maximum cap of 1000 per run.
   • sortOrder: chronological or relevance (chronological supported by API).
   • addUserInfo: Include legacy user metadata per tweet.
   • onlyUserInfo: Fetch only profile info (no tweets).
   • addNotFoundUsersToOutput / addSuspendedUsersToOutput: Emit flagged records for unavailable profiles.
   • proxyConfiguration: Optionally enable Apify Proxy; the actor auto-fallbacks to datacenter and then residential if blocked.

5. Run the actor
   Click Start. The actor will capture an authorization header dynamically, fetch timelines, and push items to the dataset as they’re collected.

6. Monitor the run
   Watch logs for progress. The proxy fallback system engages automatically on 401/403/429 responses.

7. Export results
   Open the Run dataset and export your data in JSON, CSV, or Excel for analysis, pipelines, or BI dashboards.

Pro Tip: Use onlyUserInfo for a fast inventory of profiles with legacy fields, or pass keywords in startUrls to build topic datasets without specifying usernames — a flexible path for both a Twitter profile scraper tool and X user scraper workflows.

Use cases

Use case name	Description
Marketing analytics – influencer monitoring	Track creator timelines and aggregate engagement metrics for campaign reporting and benchmarking.
Competitor intelligence – content tracking	Analyze competitor posting cadence and message themes from public timelines to inform strategy.
Lead enrichment – profile metadata capture	Use onlyUserInfo to enrich records with display name, bio, and verification status for outreach.
Social listening – keyword timelines	Collect tweets from search timelines using keywords to fuel sentiment and trend analysis.
Academic research – longitudinal studies	Build reproducible datasets from public posts for behavioral, political, or social studies.
Data engineering – API pipelines	Trigger via Apify API and stream datasets into warehouses or lakes for downstream ML/BI.
Editorial workflows – monitoring public figures	Scrape public accounts to populate dashboards for journalists and newsroom analysts.

Why choose Twitter Profile Scraper?

This production-ready Apify actor is built for precision, automation, and resilience at scale.

• ⚡ Accurate, structured outputs
  Consistent tweet objects with stable fields (id_str, full_text, entities, counts) and optional legacy user data.

• 🧱 Batch processing at scale
  Feed large lists of URLs, handles, IDs, or keywords; items are saved live for reliability.

• 🔄 Advanced proxy resilience
  Automatic fallback (none → datacenter → residential) with exponential backoff to minimize blocks.

• 💻 Developer access & API integration
  Works as a Twitter profile scraping API via Apify — integrate with Python scripts or orchestration tools.

• 🛡️ Public data focus
  Designed for public X/Twitter content only; no login or private data access.

• 💾 Flexible exports
  Download from the dataset as JSON, CSV, or Excel to fit analytics and ETL workflows.

• 🏗️ Beyond extensions
  More stable than brittle browser add-ons: Playwright-backed auth capture and robust pagination for production use.

In short, it combines reliable extraction with proxy-aware infrastructure to outperform generic scrapers and unstable extensions.

Is it legal / ethical to use Twitter Profile Scraper?

Yes — when done responsibly. This actor targets public X/Twitter data and does not access private profiles or authenticated content.

Guidelines for compliant use:

• Scrape only publicly available information and respect the platform’s Terms of Service.
• Avoid accessing private data or bypassing security measures.
• Observe rate limits and platform policies to reduce disruption.
• Ensure your use aligns with applicable data protection laws (e.g., GDPR/CCPA).
• Consult your legal team for edge cases and jurisdiction-specific requirements.

Input parameters & output format

Example JSON input

{
  "startUrls": [
    "https://x.com/elonmusk",
    "elonmusk",
    "44196397",
    "tesla"
  ],
  "maxTweets": 100,
  "sortOrder": "chronological",
  "addUserInfo": true,
  "onlyUserInfo": false,
  "addNotFoundUsersToOutput": false,
  "addSuspendedUsersToOutput": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Input parameters

• startUrls (array, required): List of Twitter profile URLs (e.g., https://x.com/username), usernames (e.g., username), user IDs (e.g., 44196397), or search keywords (e.g., tesla). Supports bulk input. Default: none.
• maxTweets (integer): Maximum number of tweets to fetch per user or keyword. Minimum: 1, Maximum: 100, Default: 10. Note: Internally, the actor enforces a minimum of 10 and a maximum cap of 1000 per run.
• sortOrder (string): Sort order for tweets (currently supports chronological order from API). Enum: ["chronological", "relevance"]. Default: "chronological".
• addUserInfo (boolean): Include user profile data (legacy) in each tweet output. Default: true.
• onlyUserInfo (boolean): If enabled, only fetch user profile information without tweets. Default: false.
• addNotFoundUsersToOutput (boolean): Include users that were not found in the output with notFound flag. Default: false.
• addSuspendedUsersToOutput (boolean): Include suspended users in the output with suspended flag. Default: false.
• proxyConfiguration (object): Proxy configuration. By default, no proxy is used. If Twitter blocks requests, the actor will automatically fallback to datacenter proxy, then residential proxy with retries. Default: {"useApifyProxy": false}.

Example JSON output (tweet with user metadata)

{
  "user": {
    "screen_name": "elonmusk",
    "name": "Elon Musk",
    "description": "Mars & Cars, Chips & Dips",
    "followers_count": 229033543,
    "friends_count": 1226,
    "statuses_count": 89153,
    "favourites_count": 182734,
    "listed_count": 165176,
    "created_at": "Tue Jun 02 20:12:29 +0000 2009",
    "verified": true,
    "profile_image_url": "https://pbs.twimg.com/profile_images/...",
    "profile_banner_url": "https://pbs.twimg.com/profile_banners/...",
    "default_profile": false,
    "default_profile_image": false,
    "entities": {
      "description": {
        "urls": []
      }
    }
  },
  "id_str": "1519480761749016577",
  "full_text": "Next I'm buying Coca-Cola to put the cocaine back in",
  "created_at": "Thu Apr 28 00:56:58 +0000 2022",
  "favorite_count": 4289223,
  "retweet_count": 594435,
  "reply_count": 170050,
  "quote_count": 167104,
  "bookmark_count": 21112,
  "conversation_id_str": "1519480761749016577",
  "user_id_str": "44196397",
  "lang": "en",
  "is_quote_status": false,
  "favorited": false,
  "retweeted": false,
  "bookmarked": false,
  "display_text_range": [0, 52],
  "entities": {
    "hashtags": [],
    "symbols": [],
    "user_mentions": [],
    "urls": [],
    "timestamps": []
  },
  "extended_entities": null
}

Additional cases

• onlyUserInfo = true (profile-only):

{
  "user": {
    "screen_name": "apify",
    "name": "Apify",
    "description": "Web scraping and automation platform",
    "created_at": "Mon Nov 16 12:00:00 +0000 2015",
    "verified": true
  }
}

• Not found or suspended (when enabled via flags):

{ "user": { "screen_name": "nonexistent_user" }, "notFound": true }

{ "user": { "screen_name": "some_user" }, "suspended": true }

FAQ

Do I need to log in to use this X profile scraper?

No. The actor captures a public authorization header dynamically with Playwright and accesses only public data, so no user login or cookies are required.

Can I scrape by keyword, or is it just a Twitter handle scraper?

You can do both. Provide a keyword in startUrls to collect tweets from public search timelines, or provide URLs/handles/user IDs to scrape specific profiles.

Is this a Twitter profile scraping API I can integrate with?

Yes. As an Apify actor, it’s API-accessible. You can programmatically start runs, poll status, and download datasets (JSON, CSV, Excel) to integrate with your pipelines.

How many tweets can I extract per user?

You control this with maxTweets (UI default 10; UI range 1–100). Internally, the actor enforces a minimum of 10 and caps at 1000 per run.

Does this work as a Twitter followers scraper?

It does not fetch follower lists. However, when addUserInfo is enabled, the legacy user object includes followers_count, which you can use for analytics.

What happens if a profile is suspended or not found?

If you enable addSuspendedUsersToOutput and/or addNotFoundUsersToOutput, the actor emits minimal objects with suspended: true or notFound: true so you can track outcomes reliably.

Can I exclude user metadata to slim the output?

Yes. Set addUserInfo to false to remove the user object from each tweet item and reduce payload size.

How does the proxy system work if requests are blocked?

The actor auto-fallbacks from no proxy → datacenter → residential with retries. It updates the proxy state and continues fetching to minimize interruptions.

Closing thoughts

Twitter Profile Scraper is built to turn public X/Twitter timelines and profiles into structured, export-ready data at scale. With resilient proxy fallback, profile-only mode, keyword support, and clean JSON outputs, it’s ideal for marketers, developers, researchers, and analysts. Trigger runs via the Apify API or your Python workflows, export to JSON/CSV/Excel, and integrate with your downstream pipeline — start extracting smarter with a reliable Twitter profile scraper tool today.