X/Twitter Tweet Scraper

An Apify Actor that scrapes tweet details and reply threads from X (Twitter) via the authenticated GraphQL API. Provide tweet URLs or IDs plus your X login cookies, and get clean, structured JSON: full text, authors, metrics, media, hashtags, and replies.

Why cookies? X now blocks anonymous (guest-token) scraping at the edge. This Actor authenticates as your logged-in X session, giving reliable access to tweet details and conversations. See Getting your X cookies.

---

What it does

Feature	Supported
Scrape a tweet by URL or ID	✅
Pull the full reply thread	✅
Structured output (author, metrics, media, hashtags, mentions)	✅
Apify Proxy integration (residential rotation)	✅
Streaming results (tweets appear live as the run progresses)	✅
Search & user-timeline modes	🚧 Coming soon

---

How to use

1. Get your X cookies

The Actor authenticates as your logged-in X account using two cookies:

1. Open https://x.com in your browser and log in.

2. Open DevTools:

   • Chrome / Edge: F12 → Application tab → Storage → Cookies → https://x.com
   • Firefox: F12 → Storage tab → Cookies → https://x.com

3. Find and copy these two cookie Values:

   Cookie name	What it is
   auth_token	Long-lived session token (~months).
   ct0	CSRF token (must be paired with it).

⚠️ Security: these cookies grant full access to your X account. Consider using a dedicated/throwaway X account for scraping.

2. Run the Actor

1. Go to the Actor page on Apify.
2. Fill in the input:
   • Tweet URLs or IDs — one or more tweet URLs or IDs
   • auth_token cookie — your auth_token value
   • ct0 cookie — your ct0 value
   • Include replies — toggle the reply thread on/off
   • Proxy configuration — leave as Apify Proxy (recommended) for IP rotation
3. Click Start and watch tweets stream into the dataset in real time.
4. Export the results as JSON, CSV, Excel, or via the API.

---

Input

Field	Type	Required	Default	Description
mode	string	yes	tweet	Scrape mode (v1: tweet only)
targets	array	yes	—	Tweet URLs (https://x.com/user/status/123) or IDs
authToken	string	yes	—	X auth_token cookie
ct0	string	yes	—	X ct0 cookie
maxTweets	integer	no	50	Max tweets per target
includeReplies	boolean	no	true	Include the reply thread
requestDelay	number	no	1.0	Seconds between page fetches
maxRetries	integer	no	5	Retries on 429/5xx
proxy	proxy	no	Apify Proxy	Proxy config (Apify Proxy recommended)

---

Output

Each tweet is pushed to the dataset as structured JSON matching this schema:

{
  "id": "1784210000000000000",
  "text": "Full text of the tweet, t.co links expanded.",
  "created_at": "2024-04-25T12:34:56+00:00",   // ISO-8601
  "author": {
    "id": "44196397",
    "username": "elonmusk",
    "display_name": "Elon Musk",
    "followers_count": 180000000,
    "is_verified": true
  },
  "metrics": {
    "likes": 12345, "retweets": 678, "replies": 90,
    "quotes": 12, "views": 987654
  },
  "media": [
    { "type": "photo", "url": "https://pbs.twimg.com/...", "alt_text": null },
    { "type": "video", "url": "https://video.twimg.com/.../vid/avc1/1280x720/...", "alt_text": null }
  ],
  "hashtags": ["AI"],
  "mentions": ["OpenAI"],
  "urls": ["https://openai.com"],
  "url": "https://x.com/elonmusk/status/1784210000000000000",
  "in_reply_to": null,             // parent tweet id if this is a reply
  "conversation_id": "1784210000000000000",
  "language": "en",
  "source": "Twitter Web App"
}

The Apify UI includes a pre-built Overview table view (author, tweet text, likes, retweets, views, timestamp, link) for quick browsing.

---

How it works (technical)

Which endpoints it targets

X's web app talks to a GraphQL API at https://x.com/i/api/graphql/<queryId>/<OperationName>. The Actor uses the TweetDetail operation, which returns the focal tweet plus its reply thread in a single response.

Authentication (cookie / CSRF flow)

• Bearer token — X's public web bearer (hardcoded in the web app's JS, identical for all sessions).
• Cookies — auth_token (session) + ct0 (CSRF) seeded into the httpx cookie jar.
• x-csrf-token header — set to the ct0 value on every request (X enforces these match).
• x-twitter-auth-type: OAuth2Session — marks the request as an authenticated web session.

A session-sanity check runs at startup so expired/invalid cookies fail fast with a clear message.

Streaming results

Each parsed tweet is pushed to the Apify dataset the moment it's built (via an on_tweet callback wired to Actor.push_data). This means:

• Results appear live in the Apify UI as the run progresses.
• A crash or timeout mid-run still preserves all tweets scraped so far.

Rate limiting & resilience

All network calls use exponential backoff with full jitter on HTTP 429 and 5xx, honouring the x-rate-limit-reset header when present. A configurable requestDelay adds politeness between requests. Per-target error isolation ensures one failed tweet or target never crashes the whole run.

---

Local development

git clone https://github.com/Coderz-devz/twitter-scraper.git
cd twitter-scraper
python -m venv .venv && source .venv/Scripts/activate  # Windows Git Bash
pip install -r requirements.txt

# Set APIFY_MEMORY_STORAGE so the SDK uses local storage, not the cloud.
export APIFY_MEMORY_STORAGE=1
python -m src.main

Deploy to Apify

npm i -g apify          # install the Apify CLI
apify login             # paste your Apify API token
apify push              # builds the Docker image and deploys the Actor

---

Refreshing query ids

1. Open https://x.com (logged in) → DevTools → Network tab.
2. Filter by TweetDetail, open any tweet.
3. Right-click the TweetDetail request → Copy URL.
4. The URL looks like .../graphql/<queryId>/TweetDetail?.... Copy the id and update QUERY_IDS["TweetDetail"] in src/scraper.py.

---

License

MIT — see LICENSE.