Twitter (X) Comments Scraper

Scrapes the replies (comments) of X/Twitter posts without the official API. Give it one or many post URLs; it opens each conversation in a logged-in browser, scrolls it, expands hidden reply sections ("Show more replies", "Show probable spam") and exports every reply as structured JSON.

How it works

The Actor reads X's own conversation (TweetDetail) responses while scrolling, so output is clean structured data rather than scraped HTML. Unrelated "Discover more" tweets are dropped using the conversation id.

Authentication

X hides replies from anonymous visitors, so a session is required. Resolved in this order:

1. cookies input (Cookie-Editor JSON export; needs auth_token and ct0)
2. TW_COOKIES secret env var (JSON array or name=value; name2=value2)
3. Session cached from a previous automated login (key-value store twitter-session)
4. Automated login with twUsername / twPassword (+ twEmail), or TW_USERNAME / TW_PASSWORD / TW_EMAIL env vars. 2FA must be off. Cookies are more reliable than automated login.

Always use a throwaway account; accounts used for scraping can be restricted.

Input

urls, startUrls, resultsPerCategory (default 100, max 1000 per post), scrapeAll, includeSourceTweet, cookies, twUsername, twPassword, twEmail, proxyConfiguration (residential recommended), headless.

Output

One item per reply, plus one for the source post (is_source_tweet: true). Every item carries metadata.sourceTweetId and inputUrl. A reply whose in_reply_to_status_id_str equals the source id is a top-level comment; otherwise it points at its parent reply.

On empty runs the Actor saves DEBUG_SCREENSHOT and DEBUG_HTML to the key-value store.

Notes

Scraping may violate X's Terms of Service; collect only public data and respect data-protection law.