Twitter X Posts Scraper

Twitter X Posts Scraper

Twitter X Posts Scraper extracts public X posts (tweets) from Twitter/X profiles at scale and in real time — then delivers clean, structured results for analysis and automation. It solves the pain of manual copy-paste by programmatically fetching tweet text, engagement metrics, media, and author info from profile timelines, with smart proxy fallback to reduce blocking. Built for marketers, developers, data analysts, and researchers, it enables social listening, brand monitoring, sentiment analysis, and research at scale — with export-ready data and configurable sorting.

What data / output can you get?

Below are real output fields pushed to the Apify dataset by this actor, with examples to illustrate the structure and values you’ll receive.

Data type	Description	Example value
id	Tweet ID (rest_id)	"1988877569597260072"
url	Canonical tweet URL	"https://x.com/elonmusk/status/1988877569597260072"
user_posted	Username (screen_name) of the author	"elonmusk"
name	Display name of the author	"Elon Musk"
description	Full tweet text	"@tetsuoai Long press on any image to turn it into a video..."
date_posted	ISO timestamp (UTC)	"2025-11-13T07:52:18.000Z"
likes	Favorite count	1729
replies	Reply count	554
reposts	Retweet/Repost count	368
quotes	Quote count	38
views	Impressions/view count	"1399060"
bookmarks	Bookmark count	213
is_verified	Blue check status	true
followers	Author followers count	229031060
following	Author following count	1226
posts_count	Total posts by the author	89153
profile_image_link	Author profile image URL	"https://pbs.twimg.com/profile_images/.../oTT3nm5Z_normal.jpg"
biography	Author bio text	""
hashtags	Array of hashtags in the tweet	null or ["AI","Tech"]
tagged_users	Array of mentioned users	["tetsuoai"]
photos	First photo URL if present	null or "https://pbs.twimg.com/media/..."
videos	Array of MP4 URLs (highest bitrate first)	["https://video.twimg.com/.../file.mp4"]
quoted_post	Object with quoted post basics	{"post_id": null, "description": null, ...}
external_url	Author’s external profile URL if present	null or "https://tesla.com"
input	Object with the constructed tweet URL	{"url":"https://x.com/elonmusk/status/1988877569597260072/"}

Notes:

• photos returns the first photo URL if present; videos returns an array of MP4 variants (sorted by bitrate).
• quoted_post contains a minimal snapshot with fields: data_posted, description, post_id, profile_id, profile_name, url, videos.
• Export your dataset to CSV, JSON, or Excel directly from Apify.

Key features

• ⚡ Dynamic authorization capture Automatically collects a valid Bearer authorization header from live Twitter/X network traffic via Playwright before API requests.

• 🧭 Robust proxy fallback Built-in fallback from no proxy → datacenter → residential (with retries) to mitigate blocks and empty responses.

• 🗂️ Structured tweet records Pushes well-formed items including text, engagement metrics (likes, replies, reposts, quotes, views, bookmarks), media (photos/videos), hashtags, mentions, and author stats.

• 🧮 Sortable results Choose sortOrder = recent, oldest, or popular to control how tweets are ordered before saving.

• 📏 Precise limits Control depth with maxTweets per input to bound runtime and cost.

• 📤 Export-ready outputs Results land in the Apify dataset for easy download as CSV, JSON, or Excel — ideal for analytics and reporting.

• 👨‍💻 Developer-friendly workflow Clean, consistent field names and live pushData make it simple to integrate into pipelines for marketing analytics, data science, or research.

• 🛡️ Production-aware reliability Defensive error handling and retry logic across ID resolution and timeline fetches to keep long runs stable.

How to use Twitter X Posts Scraper - step by step

1. Create or log in to your Apify account Open Apify and ensure you have enough platform credits/resources to run the actor.

2. Add your inputs In the startUrls field, paste one value per line. Accepted formats:

   • Profile URL: https://x.com/username or https://twitter.com/username
   • Username: username or @username Note: Tweet/status URLs are detected and skipped. Provide usernames or profile URLs for timeline scraping.

3. Choose the sort order Set sortOrder to one of:

   • recent (newest first)
   • oldest (oldest first)
   • popular (most liked first)

4. Set collection limits Define maxTweets to cap how many posts are collected per input (min 1, max 100 per schema).

5. Configure proxy (optional but recommended if blocked) Use proxyConfiguration to enable Apify Proxy. The actor can step through fallback options automatically when requests fail.

6. Start the run Click Start. The actor resolves user IDs (via API or Playwright intercept), fetches timeline data, sorts based on your preference, and streams results to the dataset.

7. Download your data Open the run’s Dataset tab and export results as CSV, JSON, or Excel for immediate analysis.

Pro Tip: If you encounter blocks, timeouts, or empty results, enable Apify Proxy in proxyConfiguration to benefit from the built-in fallback logic.

Use cases

Use case name	Description
Social listening & brand monitoring	Track brand handles to analyze replies, reposts, and quotes; detect changes in engagement and audience reaction.
Campaign performance analysis	Measure likes, reposts, and views across recent posts to compare messaging effectiveness over time.
Trend & hashtag research	Collect hashtags and mentions from target profiles to inform content strategy and topic tracking.
Competitive benchmarking	Monitor competitor timelines for post cadence, engagement ratios, and content formats (photos/videos).
Academic & policy research	Build structured datasets of public posts for sentiment analysis, discourse studies, and longitudinal research.
Data engineering pipelines	Export CSV/JSON and feed clean tweet records into ETL, BI dashboards, or Python/R ML workflows.
Influencer due diligence	Evaluate audience size and engagement metrics (followers, likes, views) before partnerships.

Why choose Twitter X Posts Scraper?

This actor is built for precision, automation, and reliable scaling on Apify’s infrastructure.

• 🎯 Accurate field mapping: Consistent JSON structure with tweet text, metrics, media, and author metadata.
• 🔁 Resilient proxy strategy: Automated fallback from no proxy to datacenter to residential proxies with retries.
• 📊 Export-ready output: One-click CSV/JSON/Excel from the dataset — no post-processing required.
• 🧩 Pipeline-friendly: Deterministic field names make it easy to plug into analytics, research, or enrichment workflows.
• 🛡️ Safer than extensions: Runs in a controlled environment with headless browser orchestration and retry logic rather than brittle browser add-ons.
• 💸 Cost-conscious limits: Control depth with maxTweets and efficient per-page fetching.

In short, it’s a production-focused Twitter/X post scraper designed for stable runs and clean, analysis-ready data.

Is it legal / ethical to use Twitter X Posts Scraper?

Yes — when used responsibly. This actor targets public Twitter/X data from profile timelines and does not access private accounts or protected content.

Compliance guidelines:

• Only collect publicly available information.
• Respect the platform’s terms and applicable laws (e.g., GDPR/CCPA where relevant).
• Avoid high-frequency request patterns that could cause undue load.
• Do not attempt to bypass access controls or scrape private/locked profiles.
• For edge cases, consult your legal team to ensure compliant use in your jurisdiction.

Input parameters & output format

Input JSON example

{
  "startUrls": [
    "elonmusk",
    "https://x.com/BarackObama",
    "@nasa",
    "https://twitter.com/github"
  ],
  "sortOrder": "recent",
  "maxTweets": 10,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Parameters

Field	Type	Required	Default	Description
startUrls	array (stringList editor)	Yes	—	What to scrape — add one value per line. Examples: profile URLs (https://x.com/username or https://twitter.com/username) or usernames (username or @username). Tip: use clear handles or full profile URLs for best results.
sortOrder	string (enum: recent, popular, oldest)	No	recent	How to order scraped tweets before saving. Options: recent (newest first), oldest (oldest first), popular (most liked first).
maxTweets	integer (min 1, max 100)	No	10	Upper limit of tweets to collect per profile/input. Higher values take longer.
proxyConfiguration	object (proxy editor)	No	—	Apify proxy settings for Twitter/X. Default: no Apify proxy. The actor can step through fallback proxies if requests fail (datacenter, then residential retries). Enable when you see blocks, timeouts, or empty results.

Notes:

• Tweet/status URLs in startUrls are detected and skipped; provide usernames or profile URLs for timeline scraping.

Output JSON example

[
  {
    "biography": "",
    "bookmarks": 213,
    "date_posted": "2025-11-13T07:52:18.000Z",
    "description": "@tetsuoai Long press on any image to turn it into a video in less than 30 seconds https://t.co/Nsp7Ba0flp",
    "external_url": null,
    "followers": 229031060,
    "following": 1226,
    "hashtags": null,
    "id": "1988877569597260072",
    "input": {
      "url": "https://x.com/elonmusk/status/1988877569597260072/"
    },
    "is_verified": true,
    "likes": 1729,
    "name": "Elon Musk",
    "photos": null,
    "posts_count": 89153,
    "profile_image_link": "https://pbs.twimg.com/profile_images/1983681414370619392/oTT3nm5Z_normal.jpg",
    "quoted_post": {
      "data_posted": null,
      "description": null,
      "post_id": null,
      "profile_id": null,
      "profile_name": null,
      "url": null,
      "videos": null
    },
    "quotes": 38,
    "replies": 554,
    "reposts": 368,
    "tagged_users": [
      "tetsuoai"
    ],
    "url": "https://x.com/elonmusk/status/1988877569597260072",
    "user_posted": "elonmusk",
    "videos": [
      "https://video.twimg.com/amplify_video/1988877511368019968/vid/avc1/576x856/34pcJSQSXqqM4JRQ.mp4?tag=23"
    ],
    "views": "1399060"
  }
]

Field notes:

• quoted_post contains minimal context for quoted tweets (data_posted, description, post_id, profile_id, profile_name, url, videos).
• photos is a single URL when present; videos is an array of MP4 URLs sorted by bitrate.

FAQ

Do I need to provide tweet URLs or usernames?

Provide usernames or profile URLs. If a tweet/status URL is detected, the actor will skip it and log a warning. Use formats like “username”, “@username”, or “https://x.com/username”.

How many tweets can I scrape per profile?

Set maxTweets between 1 and 100 per input (as defined in the input schema). Higher values take longer to complete.

What determines the order of results?

Use sortOrder to control ordering before saving: recent (newest first), oldest (oldest first), or popular (most liked first).

How does the actor handle blocks or empty responses?

It includes proxy fallback: starting with no proxy, then switching to Apify datacenter proxy, and finally to residential proxy with retries if needed.

What data fields are included in the output?

You’ll receive tweet text, engagement counts (likes, replies, reposts, quotes, views, bookmarks), media (photos/videos), hashtags, mentions, and author metadata (followers, following, posts_count, profile_image_link, biography, is_verified), plus URLs and a quoted_post object.

Does it require login to Twitter/X?

The actor dynamically captures an authorization header from X network requests via Playwright. If this capture fails, the run stops with an error message (“Failed to capture authorization header. Make sure you're logged in to Twitter/X.”). No in-actor login flow is implemented.

Can I export results to CSV or JSON?

Yes. Open the run’s Dataset and export to CSV, JSON, or Excel for immediate use in analytics and reporting workflows.

Is there a trial or subscription?

This actor is offered on a flat monthly subscription with trial minutes available, subject to Apify Store terms. Check the actor’s listing for current pricing and trial details.

Final thoughts

Twitter X Posts Scraper is built to collect clean, structured tweet data from public Twitter/X profiles at scale. With dynamic authorization capture, smart proxy fallback, and export-ready datasets, it’s ideal for marketers, developers, analysts, and researchers who need reliable social data fast. Use it to power social listening, benchmarking, and sentiment analysis pipelines — and integrate the structured outputs into your analytics stack or automation workflows to start extracting smarter insights today.