Twitter X Tweets Profiles Scraper

Twitter X Tweets Profiles Scraper

Twitter X Tweets Profiles Scraper is a purpose-built scraping actor that collects public tweets and profile context from X (Twitter) at scale. It solves the challenge of reliable tweet extraction by normalizing inputs (usernames, profile URLs, or keywords) and returning clean, structured records ready for analysis. Ideal for marketers, developers, data analysts, and researchers, this actor delivers a consistent tweet scraper + profile context workflow for social listening, trend tracking, and competitive intelligence — all in a cloud-native setup that scales.

What data / output can you get?

Below are the primary fields this actor outputs to the Apify dataset. Examples reflect real structures returned by the scraper.

Data type	Description	Example value
id	Tweet ID (string)	"1519480761749016577"
url	Canonical tweet URL	"https://x.com/elonmusk/status/1519480761749016577"
verified	Whether the author is verified	true
username	Screen name with @ prefix	"@elonmusk"
fullname	Author’s display name	"Elon Musk"
timestamp	Tweet creation time (UTC)	"Thu Apr 28 00:56:58 +0000 2022"
text	Full tweet text	"Next I'm buying Coca-Cola to put the cocaine back in"
links	URLs extracted from entities, cards, media	[]
isReply	Whether the tweet is a reply	false
likes	Favorite/like count	4289224
retweets	Retweet count	594428
user	Nested object with profile details (included unless disabled)	{"avatar":"https://pbs.twimg.com/profile_images/...","username":"elonmusk", ...}

Notes:

• media is returned as an array of objects like {"type": "photo" | "video" | "animated_gif", "url": "https://..."}.
• Additional boolean flags include isPinned, isQuote, isRetweet, plus replies and quotes counters.
• Access all results in the Apify dataset and export in your preferred format.

Key features

• 🧠 Smart input normalization Enter profile URLs, @handles, numeric user IDs, or plain keyword searches — the actor auto-detects and routes to the right extraction path.

• 🧾 Structured tweet + profile context Each record includes tweet content, engagement metrics, and (optionally) a detailed user object via the includeUserInfo flag.

• 💬 Reply control Use withReplies to include or exclude replies from your results for cleaner timelines or full conversations.

• 🎬 Media & link extraction Collect aggregated links from entities/cards and media arrays with best-available video URLs or images.

• 🔁 Resilient proxy fallback Built-in logic automatically attempts datacenter proxy and falls back to RESIDENTIAL proxies if access is blocked — keeping runs stable.

• 📥 Incremental dataset streaming Results are pushed to the dataset as they’re found (push_data), enabling long runs and smooth pipeline handoffs.

• 🧪 Headless browser automation Uses Playwright with Chromium to capture necessary tokens and timelines reliably in a cloud environment.

• 📊 Analytics-ready schema Consistent field names and types make it easy to feed BI tools, NLP pipelines, and downstream analyses.

• 🔑 No API keys required Works without API authentication, using public endpoints and browser context to collect publicly available data.

How to use Twitter X Tweets Profiles Scraper - step by step

1. Create or log in to your Apify account Access the Apify platform to run the actor in the cloud.

2. Open the actor Find “Twitter X Tweets Profiles Scraper” in the Apify Store and open the actor’s Run page.

3. Add input data In startUrls, add any mix of:

   • Profile URLs (e.g., https://x.com/elonmusk)
   • Usernames (e.g., mrbeast or @mrbeast)
   • Keywords (e.g., crypto news)

4. Configure options

   • maxTweets: Cap results per input (1–100)
   • withReplies: Include or exclude replies
   • includeUserInfo: Toggle the nested user object
   • sortOrder: newest, oldest, or popular
   • proxyConfiguration: Set proxy behavior (the actor handles fallback automatically)

5. Start the run Click Start. The actor launches a headless browser, collects session tokens, and begins extracting tweets.

6. Monitor progress Watch logs for normalization, proxy usage, and extraction status. Results are streamed to the dataset incrementally.

7. Download results Open the run’s dataset to view and export your structured tweet records for analysis or integration.

Pro tip: Chain this dataset into your analytics or enrichment workflow and schedule runs on Apify to keep timelines fresh.

Use cases

Use case name	Description
Social media analytics for brands	Track posts and engagement from brand or competitor profiles to benchmark performance and messaging.
Influencer monitoring & discovery	Collect tweets and profile metrics to evaluate potential partners and ongoing campaigns.
Trend and sentiment research	Aggregate keyword-based timelines for topic tracking, NLP, and time-series analysis.
Journalism & timeline reconstruction	Compile verified, timestamped tweets around events to support reporting and investigation.
Academic & behavioral studies	Build reproducible datasets from public posts for language modeling and social science research.
Developer pipelines & ETL	Stream structured tweet objects into data lakes or warehouses for dashboards and ML pipelines.

Why choose Twitter X Tweets Profiles Scraper?

This actor delivers production-ready reliability with structured results and robust fallback logic.

• 🎯 Precision schema: Stable, analytics-ready fields for immediate downstream use.
• ⚡ Scalable by design: Handles multiple inputs and streams results incrementally.
• 🧑‍💻 Developer-friendly: Consistent JSON structure, ideal for pipelines and programmatic consumption.
• 🛡️ Robust access: Automatic proxy fallback (datacenter → RESIDENTIAL) when requests are blocked.
• 🔍 Flexible targeting: Works with profile URLs, usernames, IDs, and keyword searches without extra setup.
• 🔒 No API keys required: Extracts publicly available data without authenticated APIs.
• ☁️ Cloud-native: Runs on Apify infrastructure for hands-off automation and scheduling.

In short: a reliable tweet scraper vs. brittle extensions or one-off scripts — built for real operational use.

Is it legal / ethical to use Twitter X Tweets Profiles Scraper?

Yes — when done responsibly. This actor extracts publicly available information and does not access private accounts, protected content, or login-required pages.

Guidelines for compliant use:

• Only process public data you have a legitimate reason to analyze.
• Review and respect Twitter/X Terms of Service.
• Comply with data protection regulations (e.g., GDPR, CCPA) where applicable.
• Avoid misuse such as spam, harassment, or unauthorized redistribution.
• If in doubt, consult your legal team for your specific use case.

Input parameters & output format

Example JSON input

{
  "startUrls": [
    "https://x.com/elonmusk",
    "mrbeast",
    "crypto news"
  ],
  "sortOrder": "newest",
  "maxTweets": 10,
  "withReplies": true,
  "includeUserInfo": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Parameters

• startUrls (array, required): List of Twitter/X profile URLs (e.g., https://x.com/username), usernames (e.g., username or @username), or search keywords. Default: none.
• sortOrder (string): How to sort the tweets. Allowed values: newest, oldest, popular. Default: "newest".
• maxTweets (integer): Maximum number of tweets to scrape per profile/keyword (1–100). Default: 10.
• withReplies (boolean): Include reply tweets in the results. Default: true.
• includeUserInfo (boolean): Include detailed user profile information with each tweet. Default: true.
• proxyConfiguration (object): Proxy settings. Actor will automatically fallback to datacenter then residential proxy if requests are blocked. Default prefill: {"useApifyProxy": false}.

Example JSON output

{
  "id": "1519480761749016577",
  "url": "https://x.com/elonmusk/status/1519480761749016577",
  "verified": true,
  "username": "@elonmusk",
  "fullname": "Elon Musk",
  "timestamp": "Thu Apr 28 00:56:58 +0000 2022",
  "text": "Next I'm buying Coca-Cola to put the cocaine back in",
  "links": [],
  "isPinned": false,
  "isQuote": false,
  "isRetweet": false,
  "isReply": false,
  "likes": 4289224,
  "replies": 170050,
  "retweets": 594428,
  "quotes": 167104,
  "user": {
    "avatar": "https://pbs.twimg.com/profile_images/...",
    "username": "elonmusk",
    "userFullName": "Elon Musk",
    "description": "",
    "location": null,
    "website": null,
    "joinDate": "Tue Jun 02 20:12:29 +0000 2009",
    "verified": true,
    "totalLikes": 182734,
    "totalTweets": 89153,
    "totalFollowing": 1226,
    "totalFollowers": 229037864,
    "url": null,
    "userId": "44196397",
    "totalMediaCount": 4225
  },
  "media": [
    {
      "type": "photo",
      "url": "https://pbs.twimg.com/media/..."
    }
  ],
  "tweetUserId": "44196397"
}

Notes:

• If includeUserInfo is false, the user object is omitted from pushed records.
• Results are pushed to the Apify dataset incrementally as each tweet is processed.

FAQ

Is Twitter X Tweets Profiles Scraper free to try?

You can run the actor on Apify and leverage the platform’s trial to evaluate performance. Usage beyond trial limits may require a paid plan.

Does it need API keys or login?

No. The actor works without API authentication and does not log in. It collects publicly accessible data using a headless browser session.

Can it search by keywords or hashtags?

Yes. Add plain-text keywords to startUrls and the actor will fetch search timelines for those terms.

Can I include or exclude replies?

Yes. Set withReplies to control whether reply tweets are included in the output.

What profile data is included?

When includeUserInfo is true, each tweet includes a user object with fields like avatar, username, userFullName, description, location, website, joinDate, verified, totalLikes, totalTweets, totalFollowing, totalFollowers, url, userId, and totalMediaCount.

How many tweets can I get per input?

Use maxTweets to cap results from 1 to 100 per profile or keyword in a single run.

How are results delivered?

Records are streamed to the Apify dataset during the run. You can open the dataset after completion to view and export the results.

How does the proxy fallback work?

If requests are blocked, the actor can automatically switch from a datacenter proxy to a RESIDENTIAL proxy group and retry, improving reliability on challenging targets.

Closing thoughts

Twitter X Tweets Profiles Scraper is built for structured, repeatable tweet and profile-context extraction at scale. With normalized inputs, robust proxy fallback, and an analytics-ready schema, it serves marketers, developers, analysts, and researchers who need reliable social data pipelines. Developers can automate runs and consume the dataset programmatically, while analysts can quickly export structured results for dashboards and models. Start extracting smarter, cleaner Twitter/X insights with a workflow that’s built for production.