Twitter (X) Comments Scraper

Scrape the replies (comments) of Twitter/X posts without the official API. Give the actor one or many post URLs and it opens each conversation, auto-scrolls it (expanding the hidden "Show more replies" sections), and exports every reply as structured JSON — text, author, likes, retweets, reply counts, timestamps and the source post each reply belongs to. Results can be downloaded as JSON, CSV, Excel or accessed via the Apify API.

The output is built for pipelines: flat items with tweet_id and user_info.* author keys, and metadata.sourceTweetId + inputUrl on every reply, so multi-post runs stay cleanly attributed to their source posts.

Features

• 🔗 Scrape by post URL — paste any https://x.com/user/status/... (or twitter.com) links.
• 📚 Many posts per run — all URLs are scraped in a single browser session; every item carries metadata.sourceTweetId + inputUrl so replies never get mixed up between posts.
• ♾ Auto-pagination — scrolls the conversation and clicks "Show more replies" / "Show probable spam" until the per-post limit is reached or the thread runs dry (up to 1,000 replies per post).
• 🧵 Thread-aware — replies-to-replies are included, with in_reply_to_status_id_str so you can rebuild the tree; the source post itself is emitted once and flagged is_source_tweet.
• 🚮 No junk — the unrelated "Discover more" tweets X injects below conversations are filtered out via the conversation id.
• 💾 Incremental results — replies are pushed to the dataset as each post finishes, so long runs never lose collected data to a timeout.
• 🔐 Flexible authentication — session cookies, secret env vars, or fully automated login with session caching.
• 🛠 Debuggable failures — on empty runs the actor saves a screenshot and the page HTML to the run's key-value store.

Use cases

• Sentiment analysis of audience reactions to specific posts
• Brand, campaign and influencer monitoring
• Community management and social listening pipelines
• Market and academic research

Quick start

1. Enter one or more post URLs.
2. Provide a session (see Authentication below) — a throwaway X account's cookies or credentials.
3. Run. Replies appear in the dataset as each post completes.

Input

Field	Type	Description
urls	array	Post URLs (https://x.com/user/status/<id>), one or many. De-duplicated by tweet id.
startUrls	array	Same URLs as request-list { "url": ... } objects — merged with urls.
category	string	What to scrape for each post — currently replies (the comments under the post).
resultsPerCategory	integer	Target replies PER post (max 1000). Default 100. The source post itself doesn't count.
parseAllResults	boolean	Accepted for backwards compatibility; items are always fully parsed.
scrapeAll	boolean	Auto-paginate until the per-post target is reached. When false only the initial page load is returned. Default true.
includeSourceTweet	boolean	Also emit one item for the post itself, flagged is_source_tweet: true. Set false to get replies only. Default true.
cookies	array	Cookies of a logged-in X session (Cookie-Editor/EditThisCookie JSON export). The auth_token + ct0 cookies carry the session.
twUsername / twPassword / twEmail	string	Credentials for automated login (2FA must be disabled). The email answers X's occasional identity check.
proxyConfiguration	object	Proxy settings. Residential proxies strongly recommended.
headless	boolean	Uncheck only for local debugging.

Authentication

X hides conversation replies from anonymous visitors, so the actor needs a session. It resolves one in this priority order:

1. the cookies input (per-run),
2. the TW_COOKIES secret env var (JSON cookie array or a name=value; name2=value2 header string),
3. the session cached in the twitter-session key-value store by a previous automated login,
4. automated login with twUsername/twPassword (or the TW_USERNAME/TW_PASSWORD/TW_EMAIL secret env vars) — the fresh session is cached for future runs.

Always use a throwaway account, never a personal one. Accounts used for scraping can be restricted by X.

Output

One dataset item per reply (plus one for the source post, flagged is_source_tweet: true):

{
    "tweet_id": "1234567890123456790",
    "id": "1234567890123456790",
    "url": "https://x.com/someuser/status/1234567890123456790",
    "text": "Reply text...",
    "full_text": "Reply text...",
    "lang": "en",
    "created_at": "2026-05-01T10:02:00.000Z",
    "createdTime": "2026-05-01T10:02:00.000Z",
    "favorites": 5,
    "retweets": 0,
    "replies": 1,
    "quotes": 0,
    "views": 321,
    "screen_name": "someuser",
    "user_info": {
        "rest_id": "44196397",
        "name": "Some User",
        "screen_name": "someuser",
        "url": "https://x.com/someuser",
        "avatar": "https://pbs.twimg.com/profile_images/....jpg",
        "followers": 1200,
        "verified": false,
        "blue_verified": true
    },
    "user_info.name": "Some User",
    "user_info.screen_name": "someuser",
    "user_info.rest_id": "44196397",
    "user_info.avatar": "https://pbs.twimg.com/profile_images/....jpg",
    "is_reply": true,
    "in_reply_to_status_id_str": "1234567890123456789",
    "conversation_id_str": "1234567890123456789",
    "is_source_tweet": false,
    "category": "replies",
    "inputUrl": "https://x.com/author/status/1234567890123456789",
    "metadata": {
        "sourceTweetId": "1234567890123456789",
        "sourceTweetUrl": "https://x.com/author/status/1234567890123456789",
        "category": "replies"
    },
    "metadata.sourceTweetId": "1234567890123456789",
    "hashtags": [],
    "mentions": ["author"],
    "urls": [],
    "media": [],
    "scrapedAt": "2026-07-03T12:00:00.000Z"
}

Notes for consumers:

• Attribution in multi-post runs: use metadata.sourceTweetId (emitted both as a nested object and a flat dotted key, so both access styles work) or inputUrl.
• Dropping the source post: filter out items where tweet_id equals the source id, or simply where is_source_tweet is true.
• Top-level comment vs reply-to-reply: a reply whose in_reply_to_status_id_str equals the source id is a top-level comment; any other value points at the parent reply.

Usage via API

Run the actor from your own code and read the replies from the default dataset — for example with the JavaScript client:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });

const run = await client.actor('<username>/twitter-comments-scraper').call({
    urls: [
        'https://x.com/user/status/1234567890123456789',
        'https://x.com/other/status/9876543210987654321',
    ],
    resultsPerCategory: 100,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const replies = items.filter((item) => !item.is_source_tweet);

Or with plain HTTP:

curl -X POST "https://api.apify.com/v2/acts/<username>~twitter-comments-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://x.com/user/status/1234567890123456789"], "resultsPerCategory": 50}'

FAQ

How many posts can I scrape in one run? There is no fixed limit — all URLs are processed sequentially in one browser session. For very large batches, split the URLs across a few runs so a single run stays well under its timeout; replies are pushed per post, so partial runs still keep everything scraped so far.

Why do I need cookies or an account? X hides conversation replies from anonymous visitors. Any active session of a (throwaway) account is enough — the actor never posts, likes or follows.

Are replies-to-replies included? Yes. Use in_reply_to_status_id_str to rebuild the thread: when it equals the source post id the item is a top-level comment, otherwise it points at the parent reply.

Does it also return retweets or quotes? No — this actor is focused on replies (comments). The category field is reserved for possible future categories.

Troubleshooting

Symptom	Likely cause & fix
Run fails with "Could not load the X post page"	The proxy exit IP can't reach X. Use Apify Proxy with the RESIDENTIAL group (datacenter IPs are widely blocked).
Run fails with "X blocked the conversation"	No/expired session. Refresh the cookies input or TW_COOKIES, or provide twUsername/twPassword for automated login.
0 results but the run succeeded	Post deleted/protected/age-restricted, or the session expired mid-run. Check DEBUG_SCREENSHOT and DEBUG_HTML in the run's key-value store.
Fewer replies than expected	X often hides part of a conversation ("Show more replies" is capped for low-quality/spam replies) — the actor expands what X exposes to a logged-in browser.

Running locally

npm install
APIFY_LOCAL_STORAGE_DIR=./storage node src/main.js

Put the run input into ./storage/key_value_stores/default/INPUT.json.

Support & feedback

If a run misbehaves, open an issue on the actor's Issues tab in Apify Console with the run URL — the debug screenshot and HTML saved by failed runs usually pinpoint the cause quickly. Feature requests are welcome there too.

Disclaimer

Scraping publicly available data is generally legal, but you are responsible for complying with X's Terms of Service, applicable data-protection laws (GDPR, CCPA) and the rights of the platform's users. Use throwaway accounts and residential proxies, and scrape responsibly.