Twitter X Posts Scraper

Twitter X Posts Scraper

Twitter X Posts Scraper is a purpose-built tweet scraper that collects posts from public Twitter/X profiles at scale. It solves the hassle of manually copying timelines by automatically extracting tweet text, timestamps, engagement metrics, author details, and media into structured datasets. Designed for marketers, developers, data analysts, and researchers, this Twitter posts scraper helps you scrape X user timeline data reliably and export it for analysis or automation—an effective Twitter API scraping alternative for profile timelines.

What data / output can you get?

The scraper saves each tweet as a structured JSON record with rich post and author metadata. Below are the primary fields pushed to the dataset.

Data field	Description	Example value
id	Tweet/post ID (rest_id)	"1988877569597260072"
url	Canonical tweet URL	"https://x.com/elonmusk/status/1988877569597260072"
user_posted	Username (screen name) of the author	"elonmusk"
name	Display name of the author	"Elon Musk"
description	Full tweet text	"@tetsuoai Long press on any image to turn it into a video..."
date_posted	ISO timestamp (UTC) when the tweet was created	"2025-11-13T07:52:18.000Z"
likes	Like count	1729
replies	Reply count	554
reposts	Retweet/Repost count	368
quotes	Quote count	38
views	View/impression count	1399060
bookmarks	Bookmark count	213
is_verified	Whether the account is blue-verified	true
followers	Author followers count	229031060
following	Author following count	1226
posts_count	Total tweets by the author	89153
profile_image_link	Profile image URL	"https://pbs.twimg.com/profile_images/.../normal.jpg"
biography	Author bio (when available)	""
hashtags	List of hashtags found in the tweet text	["AI","video"]
tagged_users	User mentions in the tweet	["tetsuoai"]
photos	First photo URL if present (string)	"https://pbs.twimg.com/media/....jpg"
videos	Array of video URLs (best quality first)	["https://video.twimg.com/.../file.mp4"]
quoted_post	Metadata for a quoted tweet (object)	{"post_id":"...","profile_id":"..."}
external_url	External URL set on the author’s profile	null
input	Source reference object (contains url)	{"url":"https://x.com/elonmusk/status/.../"}

Notes:

• Bonus fields include media arrays and a nested quoted_post object with rich context.
• You can export your dataset to JSON, CSV, or Excel directly from Apify.

Key features

• 🚦 Dynamic authorization capture Automatically collects a valid Authorization header from x.com network traffic to access public timeline data—no hard-coded tokens.

• 🛡️ Proxy fallback resilience Built-in proxy fallback steps through no proxy → datacenter → residential (with retries) to reduce blocks and timeouts during scraping.

• 🧭 Profile timeline focus Optimized to scrape X user timelines from usernames or user IDs, returning clean tweet objects with engagement, media, and author stats.

• 🔢 Sortable results Choose sortOrder: recent (newest first), oldest (oldest first), or popular (most liked first) to organize results before saving.

• 🎯 Per-profile limits Control crawl depth with maxTweets (1–100 per input) to tailor output size and runtime.

• 🧾 Clean, typed JSON Outputs consistent fields for downstream pipelines—perfect for no-code Twitter scraper workflows, analytics, and dashboards.

• 🔗 Flexible inputs Accepts plain strings like "elonmusk" or "@username", full profile URLs, and object entries with url properties for bulk lists.

• 📤 Easy exports & integrations Use the Apify Dataset to download Twitter to CSV, JSON, or Excel, or connect programmatically via the Apify API in Python and other stacks.

• ⚙️ Developer-friendly Runs on the Apify platform with Python under the hood—ideal for devs building an X scraper Python pipeline or integrating into ETL workflows.

• 🏗️ Production-ready infrastructure Headless automation + HTTP API mix, live dataset writes, and proxy options make it a reliable Twitter scraping tool for recurring jobs.

How to use Twitter X Posts Scraper - step by step

1. Create or log in to your Apify account You’ll run the actor from the Apify platform and manage outputs via the Dataset.

2. Open the Twitter X Posts Scraper Find it in the Apify Store and click Try for free.

3. Add inputs in startUrls Provide one value per line. Accepted formats:

   • Username: elonmusk or @elonmusk
   • Profile URL: https://x.com/elonmusk or https://twitter.com/elonmusk
   • Object entries: { "url": "https://x.com/elonmusk" } Important: If you pass a tweet/status URL, it is detected and skipped. Provide a username or user ID instead.

4. Configure sorting and limits

   • sortOrder: recent, oldest, or popular
   • maxTweets: set how many tweets to collect per profile (1–100)

5. (Optional) Configure proxyConfiguration Leave defaults for open access. If you encounter blocks or empty results, enable Apify Proxy. The scraper can automatically fall back to datacenter and residential proxies when needed.

6. Start the run The actor resolves each username to a user ID, fetches timeline pages, transforms them into structured tweet objects, and saves records live to the Dataset.

7. Review and export results Open the run’s Dataset to preview data. Export to JSON, CSV, or Excel, or consume via the Apify API for automation.

Pro tip: Chain this X posts scraper into your analytics stack to auto-refresh timelines and feed a BI dashboard or sentiment model via the Dataset API.

Use cases

Use case	Description
Social listening & brand monitoring	Track replies, quotes, and hashtags in author timelines to quantify engagement and craft content strategy.
Competitor content analysis	Compare post frequency, media usage, and popular tweets across competitors’ timelines for actionable insights.
Campaign reporting	Export Twitter to CSV for KPIs like likes, replies, reposts, quotes, and views to share with stakeholders.
Academic & policy research	Build longitudinal datasets of public posts for topic analysis and timelines without manual collection.
Data enrichment for AI/NLP	Generate structured corpora (text + engagement + hashtags) for topic modeling and sentiment training.
Automation & ETL pipelines	Use the Apify API to pull timeline data into warehouses and trigger downstream transformations.
Influencer vetting	Assess authenticity signals such as is_verified, followers, and posts_count before collaborations.

Why choose Twitter X Posts Scraper?

Built for precision and reliability, this Twitter data scraper focuses on profile timelines and delivers structured JSON ready for analysis or automation.

• 🎯 Accurate, structured output: Clean field mapping for tweets, media, engagement, and author stats—no manual cleanup.
• 🛡️ Robust against blocks: Proxy fallback from no proxy to datacenter and residential with retries to stabilize runs.
• ⚡ Scalable & efficient: Batch multiple usernames and control per-profile limits to manage speed and cost.
• 💻 Developer-first: Works seamlessly with Apify’s API and Python ecosystem for hands-free pipelines.
• 🔌 Easy exports & integrations: Download CSV/JSON/Excel or connect to tools and internal workflows.
• 🧭 Clear scope vs. extensions: Unlike fragile browser plugins, this production-grade Twitter scraping tool focuses on stable timeline extraction.
• ✅ Ethical by design: Intended for public data access; you control scope and compliance.

In short, if you need a dependable X posts scraper for timeline data—rather than a brittle extension—this is a strong, automation-ready choice.

Is it legal / ethical to use Twitter X Posts Scraper?

Yes—when used responsibly. The scraper targets publicly available Twitter/X profile timelines and does not access private or authenticated content.

Guidelines for compliant use:

• Scrape public information only; do not target private or protected accounts.
• Respect Twitter/X Terms of Service and applicable laws (e.g., GDPR, CCPA).
• Avoid excessive request rates and be mindful of platform load and blocking.
• Do not misuse or resell data in ways that violate user privacy or regulations.
• Consult your legal team for edge cases or downstream usage policies.

Input parameters & output format

Example JSON input

{
  "startUrls": [
    "elonmusk",
    "@BarackObama",
    "https://x.com/elonmusk",
    { "url": "https://twitter.com/nasa" }
  ],
  "sortOrder": "recent",
  "maxTweets": 25,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Parameters

• startUrls

  • Type: array (string list editor; strings or objects with url)
  • Required: Yes
  • Description: What to scrape — add one value per line. Examples: https://x.com/username, https://twitter.com/username, username, or @username. Tip: use clear handles or full profile URLs for best results.
  • Default: None (must provide at least one value)

• sortOrder

  • Type: string (enum: recent, popular, oldest)
  • Required: No
  • Description: How to order scraped tweets before saving.
  • Default: "recent"

• maxTweets

  • Type: integer (minimum: 1, maximum: 100)
  • Required: No
  • Description: Upper limit of tweets to collect per profile/input. Higher values take longer.
  • Default: 10

• proxyConfiguration

  • Type: object (proxy editor)
  • Required: No
  • Description: Apify proxy settings for Twitter/X. Default: no Apify proxy. The actor can step through fallback proxies if requests fail (datacenter, then residential retries). Enable when you see blocks, timeouts, or empty results.
  • Default: { "useApifyProxy": false }

Important behavior:

• Tweet/status URLs are detected and skipped. Provide usernames or user IDs for scraping.

Example JSON output

[
  {
    "id": "1988877569597260072",
    "url": "https://x.com/elonmusk/status/1988877569597260072",
    "user_posted": "elonmusk",
    "name": "Elon Musk",
    "description": "@tetsuoai Long press on any image to turn it into a video in less than 30 seconds https://t.co/Nsp7Ba0flp",
    "date_posted": "2025-11-13T07:52:18.000Z",
    "likes": 1729,
    "replies": 554,
    "reposts": 368,
    "quotes": 38,
    "views": 1399060,
    "bookmarks": 213,
    "is_verified": true,
    "followers": 229031060,
    "following": 1226,
    "posts_count": 89153,
    "profile_image_link": "https://pbs.twimg.com/profile_images/1983681414370619392/oTT3nm5Z_normal.jpg",
    "biography": "",
    "hashtags": null,
    "tagged_users": ["tetsuoai"],
    "photos": null,
    "videos": [
      "https://video.twimg.com/amplify_video/1988877511368019968/vid/avc1/576x856/34pcJSQSXqqM4JRQ.mp4?tag=23"
    ],
    "quoted_post": {
      "data_posted": null,
      "description": null,
      "post_id": null,
      "profile_id": null,
      "profile_name": null,
      "url": null,
      "videos": null
    },
    "external_url": null,
    "input": {
      "url": "https://x.com/elonmusk/status/1988877569597260072/"
    }
  }
]

Notes:

• Some fields can be null when unavailable (e.g., hashtags, tagged_users, photos, videos, quoted_post, external_url).
• photos is a single string URL when a photo exists; videos is an array of MP4 URLs sorted by bitrate (highest first).

FAQ

Do I need to provide tweet URLs or usernames?

Provide usernames or user IDs. If you pass a tweet/status URL, the actor detects it and skips that entry. Valid inputs include "elonmusk", "@elonmusk", or full profile URLs like "https://x.com/elonmusk".

Can it scrape tweets by hashtag or via search?

No. This actor focuses on profile timelines. It extracts hashtags present in each tweet but does not scrape Twitter search results or filter by hashtag in this version of the X posts scraper.

Is login required to run the scraper?

The actor attempts to capture an authorization header dynamically from x.com. If header capture fails, it will error with “Failed to capture authorization header. Make sure you're logged in to Twitter/X.” Enabling proxies can help improve reliability.

How many tweets can I extract per profile?

You can collect between 1 and 100 tweets per input using maxTweets. Combine multiple usernames in startUrls for batch timeline scraping with this Twitter posts scraper.

What sort orders are supported?

Use sortOrder to return results as recent (newest first), oldest (oldest first), or popular (most liked first). This helps tailor timeline analysis and reporting.

Does it handle media like photos and videos?

Yes. The tweet scraper captures photos (single URL when present) and videos (an array of MP4 URLs), along with engagement stats and author information.

How do I export data to CSV or Excel?

Open the run’s Dataset in Apify and use the built-in Export options to download JSON, CSV, or Excel. This makes it a practical Twitter to CSV scraper for reporting and analytics.

How does proxy fallback work?

If a request fails due to blocks/timeouts, the tool can step through proxy modes: start without proxy, try a datacenter proxy, then retry residential proxies. This increases success rates for your Twitter data scraper runs.

Closing CTA / Final thoughts

Twitter X Posts Scraper is built to extract structured timeline data from public Twitter/X profiles at scale. With dynamic auth capture, proxy fallback, sortable outputs, and clean JSON records, it’s ideal for marketers, developers, researchers, and data teams. Export to CSV/JSON/Excel, or integrate via the Apify API to automate pipelines in your stack. Start extracting smarter profile timelines today with a reliable, no-code Twitter scraping tool designed for analysis and automation.