Twitter X Posts Scraper

Twitter X Posts Scraper

Twitter X Posts Scraper is a high-performance Twitter/X posts scraper that collects public tweets from user timelines by username, profile URL, or user ID. It removes the manual grind of copy-pasting and turns posts into clean, structured records you can analyze and automate. Built for marketers, developers, data analysts, and researchers, this X posts scraper (aka Twitter posts scraper or tweet scraper tool) supports sorting, per-profile limits, and resilient proxy fallback — enabling monitoring and analysis at scale. 🚀

What data / output can you get?

Here are the exact fields this Twitter data extractor saves to the Apify dataset for each tweet:

Data type	Description	Example value
id	Tweet REST ID	"1988877569597260072"
url	Canonical Tweet URL	"https://x.com/elonmusk/status/1988877569597260072"
user_posted	Author’s screen name (handle without @)	"elonmusk"
name	Author’s display name	"Elon Musk"
description	Full tweet text	"Hello world"
date_posted	UTC timestamp in ISO format	"2025-11-13T07:52:18.000Z"
likes	Like (favorite) count	1729
replies	Reply count	554
reposts	Retweet/Repost count	368
quotes	Quote count	38
views	View count (impressions)	1399060
bookmarks	Bookmark count	213
is_verified	Blue verification status	true
followers	Author’s followers	229031060
following	Author’s following	1226
posts_count	Author’s total post count	89153
profile_image_link	Author’s profile image URL	"https://pbs.twimg.com/profile_images/…/photo.jpg"
biography	Author’s bio	"Technologist"
hashtags	Extracted hashtags	["AI","Productivity"]
tagged_users	Mentioned users	["tetsuoai"]
photos	First photo URL (if any)	"https://pbs.twimg.com/media/ABC123.jpg"
videos	Sorted MP4 variants (highest bitrate first)	["https://video.twimg.com/.../file1.mp4"]
quoted_post	Summary of quoted post (if present)	{"data_posted":"2025-11-13T07:51:00.000Z","description":"…","post_id":"1988877000000000000","profile_id":"44196397","profile_name":"Elon Musk","url":null,"videos":null}
external_url	Author’s profile external URL	"https://tesla.com"
input	Convenience object with derived tweet URL	{"url":"https://x.com/elonmusk/status/1988877569597260072/"}

Notes:

• Some fields can be null when not present (e.g., hashtags, photos, videos, quoted_post subfields).
• Results are saved to the Apify dataset for easy programmatic access and downloads — perfect when you want to export Twitter posts to CSV or JSON.

Key features

• ⚖️ Bold sorting controls (recent/oldest/popular)
  Use the sortOrder parameter to order results as recent (newest first), oldest (oldest first), or popular (most liked first) before saving.

• 🎯 Per-user tweet limits
  Control volume with maxTweets (1–100) per input, keeping runs fast and focused for any X post extractor workflow.

• 🧠 Smart input handling
  Paste profile URLs (x.com or twitter.com), plain usernames, @handles, or numeric user IDs. Tweet/status URLs are detected and skipped with a helpful log notice.

• 🔄 Cursor pagination & clean transformation
  Robust pagination fetches multiple pages and transforms results into a consistent, analysis-ready structure.

• 🖼️ Media and entities extraction
  Capture hashtags, tagged users, photos, and sorted MP4 video URLs (highest bitrate first) from timelines — handy for a Twitter post downloader setup.

• 🧵 Quoted post snapshot
  Get key details from quoted posts (IDs, author info, timestamp, and text) when present.

• 🛡️ Resilient proxy fallback
  Automatic escalation from no proxy → datacenter → residential (with retries) improves stability for real-time scraping with an X user timeline scraper.

• 🗃️ Live dataset pushes
  Tweets are pushed to the dataset as they’re processed, making it a practical X posts downloader workflow for automation.

• 🐍 Developer-friendly (Python on Apify)
  Implemented in Python and designed for Apify’s datasets and APIs — ideal if you’re looking for a Twitter scraper Python workflow without relying on official API keys.

How to use Twitter X Posts Scraper - step by step

1. Sign in to Apify
   Open the actor on the Apify platform to configure a run.

2. Add your inputs in startUrls
   Provide one value per line. Accepted formats:

   • Profile URL: https://x.com/username or https://twitter.com/username
   • Username: username or @username
   • User ID: a numeric ID (e.g., 44196397)
     Note: Tweet/status URLs are detected and skipped.

3. Choose sort order
   Set sortOrder to recent, oldest, or popular to control how tweets are ordered in the final dataset.

4. Set a tweet limit
   Configure maxTweets (1–100) per input to match your needs.

5. Configure proxy (optional)
   Use proxyConfiguration if you see blocks, timeouts, or empty results. The scraper can fall back from no proxy to datacenter and then to residential when requests fail.

6. Start the run
   The actor resolves user IDs, paginates through timelines, orders results as requested, and streams structured records into the dataset.

7. Review and download results
   Open the run’s dataset to view all items and export Twitter posts to CSV or JSON for analytics and downstream pipelines.

Pro Tip: Add multiple usernames/URLs in startUrls to process accounts in one run. For maximum stability, enable Apify proxies if you encounter blocking.

Use cases

Use case name	Description
Social media analytics & reporting	Track engagement metrics (likes, replies, reposts, quotes, views) across user timelines to benchmark content performance.
Brand monitoring & community management	Capture posts in real time and extract hashtags, tagged users, and quoted contexts for faster responses.
Market research for campaigns	Analyze posting cadence and themes from influencer or competitor timelines to inform messaging and creative decisions.
Academic & journalism research	Build clean datasets of public tweets for reproducible analyses and investigative timelines.
Developer pipelines (API/datasets)	Feed structured tweet records into downstream systems for enrichment, dashboards, or ML preprocessing.
Content and media tracking	Collect photos and MP4 variants with attributed entities for archives, audits, or creative reuse checks.

Why choose Twitter X Posts Scraper?

A precise, production-ready Twitter posts scraper that prioritizes accuracy, stability, and structured outputs.

• 🧠 Structured fields out of the box: Clean JSON aligned to consistent keys for easy parsing.
• 🛡️ Robust proxy fallback: No proxy → datacenter → residential (with retries) keeps jobs moving under pressure.
• ⚙️ Sorting & limits that fit your workflow: Control result order and per-profile volume precisely.
• 🖼️ Media-aware extraction: Photos, MP4 video URLs, hashtags, mentions, and quoted post snapshots when available.
• 🧰 Developer-ready on Apify: Works seamlessly with Apify datasets and APIs for automation pipelines.
• 🧩 Web-request approach: A Twitter scraper without API keys, using dynamic header capture and resilient pagination.
• 🔄 More reliable than extensions: Server-side actor execution avoids browser add-on instability.

Bottom line: a focused X posts scraper that delivers the tweet fields you need, in the format you expect, with resilience built in.

Is it legal / ethical to use Twitter X Posts Scraper?

Yes—when used responsibly. This actor extracts publicly available Twitter/X data and does not access private profiles or non-public endpoints.

Guidelines for responsible use:

• Only collect publicly available information.
• Respect Twitter/X terms and applicable laws (e.g., GDPR/CCPA).
• Avoid scraping private or protected accounts.
• Use proxies and request volumes responsibly.
• Consult your legal team for jurisdiction-specific constraints.

Input parameters & output format

Example JSON input

{
  "startUrls": [
    "elonmusk",
    "https://x.com/BarackObama",
    "@username",
    "44196397"
  ],
  "sortOrder": "recent",
  "maxTweets": 25,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Input reference

• startUrls (array)
  Description: “Twitter URLs, Usernames, or Keywords” — add one value per line. Examples: https://x.com/username, https://twitter.com/username, username, @username, or a numeric user ID. Tip: use clear handles or full profile URLs for best results.
  Default: none (prefill example: "elonmusk")
  Required: Yes

• sortOrder (string; enum: recent, popular, oldest)
  Description: How to order scraped tweets before saving.
  Default: "recent"
  Required: No

• maxTweets (integer; min: 1, max: 100)
  Description: Upper limit of tweets to collect per profile/input. Higher values take longer.
  Default: 10
  Required: No

• proxyConfiguration (object)
  Description: Apify proxy settings for Twitter/X. Default: no Apify proxy. The actor can step through fallback proxies if requests fail (datacenter, then residential retries). Enable when you see blocks, timeouts, or empty results.
  Default: {"useApifyProxy": false}
  Required: No

Example JSON output

[
  {
    "biography": "Technologist",
    "bookmarks": 213,
    "date_posted": "2025-11-13T07:52:18.000Z",
    "description": "Hello world",
    "external_url": null,
    "followers": 229031060,
    "following": 1226,
    "hashtags": ["AI", "Productivity"],
    "id": "1988877569597260072",
    "input": {
      "url": "https://x.com/elonmusk/status/1988877569597260072/"
    },
    "is_verified": true,
    "likes": 1729,
    "name": "Elon Musk",
    "photos": "https://pbs.twimg.com/media/ABC123.jpg",
    "posts_count": 89153,
    "profile_image_link": "https://pbs.twimg.com/profile_images/…/photo.jpg",
    "quoted_post": {
      "data_posted": null,
      "description": null,
      "post_id": null,
      "profile_id": null,
      "profile_name": null,
      "url": null,
      "videos": null
    },
    "quotes": 38,
    "replies": 554,
    "reposts": 368,
    "tagged_users": ["tetsuoai"],
    "url": "https://x.com/elonmusk/status/1988877569597260072",
    "user_posted": "elonmusk",
    "videos": [
      "https://video.twimg.com/…/file1.mp4"
    ],
    "views": 1399060
  ]
]

Notes:

• Fields like hashtags, photos, videos, and quoted_post subfields may be null when not present in the source tweet.
• The actor skips tweet/status URLs in input; provide usernames, profile URLs, or user IDs.

FAQ

Does this Twitter X Posts Scraper work with usernames, profile URLs, and user IDs?

Yes. You can add usernames (with or without @), full profile URLs from x.com or twitter.com, and numeric user IDs in startUrls. The actor normalizes these formats automatically.

Can I scrape specific tweet URLs, threads, or hashtag searches?

No. Tweet/status URLs are detected and skipped, and this tool does not perform keyword or hashtag scraping. It’s optimized as an X user timeline scraper that collects posts from profiles, not a Twitter thread scraper or Twitter hashtag scraper.

How many tweets can I collect per user?

You can set maxTweets between 1 and 100 per input. The actor paginates and stops when it reaches your limit or when no more pages are available.

How are results ordered?

Use sortOrder to choose recent (newest first), oldest (oldest first), or popular (most liked first). The actor sorts the final list before saving.

What engagement and media fields are included?

The output includes likes, replies, reposts, quotes, views, bookmarks (when available), plus hashtags, tagged users, photos, and MP4 video URLs. It also captures a summarized quoted_post object when relevant.

How does proxy handling work?

If requests fail, the scraper can automatically fall back from no proxy to a datacenter proxy, and then to residential proxies with retries. You can also set proxyConfiguration upfront to improve stability.

Is this a Twitter scraper without API keys, and can I use it with Python?

Yes. The actor captures an authorization header dynamically via Playwright and uses web requests, so you don’t need Twitter API keys. It’s implemented in Python and runs on Apify; you can trigger and integrate runs via Apify APIs in your Python workflows.

Can I export the data to CSV or JSON?

Yes. Results are saved to the run’s Apify dataset as structured JSON objects, which you can download or export (e.g., export Twitter posts to CSV) directly from the Apify UI or via API.

Closing CTA / Final thoughts

Twitter X Posts Scraper is built to turn public timelines into structured, reliable tweet data for analysis and automation. With normalized inputs, ordering options, resilient proxy fallback, and rich media/engagement fields, it helps marketers, developers, analysts, and researchers collect Twitter posts with confidence. Use Apify datasets and APIs to plug this X posts scraper into your pipelines — and start extracting smarter, cleaner Twitter/X data today.

What are other Twitter scraping tools?

If you want to scrape specific Twitter data, you can use any of the dedicated scrapers below for faster and more targeted results.

Scraper Name	Scraper Name
GetDayTrends Worldwide Twitter Trends	Twitter X Tweets Profiles Scraper
Twitter Profile Scraper	Twitter X Tweets Scraper
Twitter User Following Scraper	X Twitter Posts Search
Twitter X Reply Scraper