# X/Twitter Tweet Scraper (`api_heros/twitter-scraper`) Actor

Scrape tweet details and reply threads from X (Twitter) via the authenticated GraphQL API. Extract full tweet text, authors, metrics, media, hashtags, and replies as structured JSON.

- **URL**: https://apify.com/api\_heros/twitter-scraper.md
- **Developed by:** [API HEROS](https://apify.com/api_heros) (community)
- **Categories:** Social media, Lead generation
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.20 / 1,000 actor starts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## X/Twitter Tweet Scraper

An [Apify Actor](https://apify.com/actors) that scrapes **tweet details and reply threads** from X (Twitter) via the authenticated GraphQL API. Provide tweet URLs or IDs plus your X login cookies, and get clean, structured JSON: full text, authors, metrics, media, hashtags, and replies.

> **Why cookies?** X now blocks anonymous (guest-token) scraping at the edge. This Actor authenticates as your logged-in X session, giving reliable access to tweet details and conversations. See [Getting your X cookies](#getting-your-x-cookies).

***

### What it does

| Feature                                                        | Supported |
|----------------------------------------------------------------|---------- |
| Scrape a tweet by URL or ID                                    | ✅        |
| Pull the full reply thread                                     | ✅        |
| Structured output (author, metrics, media, hashtags, mentions) | ✅        |
| Apify Proxy integration (residential rotation)                 | ✅        |
| Streaming results (tweets appear live as the run progresses)   | ✅ |
| Search & user-timeline modes                                   | 🚧 Coming soon |

***

### How to use

#### 1. Get your X cookies

The Actor authenticates as **your logged-in X account** using two cookies:

1. Open <https://x.com> in your browser and **log in**.
2. Open DevTools:
   - **Chrome / Edge**: `F12` → **Application** tab → **Storage** → **Cookies** → `https://x.com`
   - **Firefox**: `F12` → **Storage** tab → **Cookies** → `https://x.com`
3. Find and copy these two cookie **Values**:

   | Cookie name  | What it is                          |
   |--------------|-------------------------------------|
   | `auth_token` | Long-lived session token (~months). |
   | `ct0`        | CSRF token (must be paired with it).|

> ⚠️ **Security:** these cookies grant full access to your X account. Consider using a dedicated/throwaway X account for scraping.

#### 2. Run the Actor

1. Go to the Actor page on Apify.
2. Fill in the input:
   - **Tweet URLs or IDs** — one or more tweet URLs or IDs
   - **auth\_token cookie** — your `auth_token` value
   - **ct0 cookie** — your `ct0` value
   - **Include replies** — toggle the reply thread on/off
   - **Proxy configuration** — leave as Apify Proxy (recommended) for IP rotation
3. Click **Start** and watch tweets stream into the dataset in real time.
4. Export the results as **JSON**, **CSV**, **Excel**, or via the **API**.

***

### Input

| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `mode` | string | yes | `tweet` | Scrape mode (v1: `tweet` only) |
| `targets` | array | yes | — | Tweet URLs (`https://x.com/user/status/123`) or IDs |
| `authToken` | string | yes | — | X `auth_token` cookie |
| `ct0` | string | yes | — | X `ct0` cookie |
| `maxTweets` | integer | no | `50` | Max tweets per target |
| `includeReplies` | boolean | no | `true` | Include the reply thread |
| `requestDelay` | number | no | `1.0` | Seconds between page fetches |
| `maxRetries` | integer | no | `5` | Retries on 429/5xx |
| `proxy` | proxy | no | Apify Proxy | Proxy config (Apify Proxy recommended) |

***

### Output

Each tweet is pushed to the dataset as structured JSON matching this schema:

```jsonc
{
  "id": "1784210000000000000",
  "text": "Full text of the tweet, t.co links expanded.",
  "created_at": "2024-04-25T12:34:56+00:00",   // ISO-8601
  "author": {
    "id": "44196397",
    "username": "elonmusk",
    "display_name": "Elon Musk",
    "followers_count": 180000000,
    "is_verified": true
  },
  "metrics": {
    "likes": 12345, "retweets": 678, "replies": 90,
    "quotes": 12, "views": 987654
  },
  "media": [
    { "type": "photo", "url": "https://pbs.twimg.com/...", "alt_text": null },
    { "type": "video", "url": "https://video.twimg.com/.../vid/avc1/1280x720/...", "alt_text": null }
  ],
  "hashtags": ["AI"],
  "mentions": ["OpenAI"],
  "urls": ["https://openai.com"],
  "url": "https://x.com/elonmusk/status/1784210000000000000",
  "in_reply_to": null,             // parent tweet id if this is a reply
  "conversation_id": "1784210000000000000",
  "language": "en",
  "source": "Twitter Web App"
}
```

The Apify UI includes a pre-built **Overview** table view (author, tweet text,
likes, retweets, views, timestamp, link) for quick browsing.

***

### How it works (technical)

#### Which endpoints it targets

X's web app talks to a GraphQL API at
`https://x.com/i/api/graphql/<queryId>/<OperationName>`. The Actor uses the
**`TweetDetail`** operation, which returns the focal tweet plus its reply
thread in a single response.

#### Authentication (cookie / CSRF flow)

- **Bearer token** — X's public web bearer (hardcoded in the web app's JS,
  identical for all sessions).
- **Cookies** — `auth_token` (session) + `ct0` (CSRF) seeded into the httpx
  cookie jar.
- **`x-csrf-token` header** — set to the `ct0` value on every request (X
  enforces these match).
- **`x-twitter-auth-type: OAuth2Session`** — marks the request as an
  authenticated web session.

A session-sanity check runs at startup so expired/invalid cookies fail fast
with a clear message.

#### Streaming results

Each parsed tweet is pushed to the Apify dataset the moment it's built (via an
`on_tweet` callback wired to `Actor.push_data`). This means:

- Results appear **live** in the Apify UI as the run progresses.
- A crash or timeout mid-run still preserves all tweets scraped so far.

#### Rate limiting & resilience

All network calls use exponential backoff with full jitter on HTTP 429 and
5xx, honouring the `x-rate-limit-reset` header when present. A configurable
`requestDelay` adds politeness between requests. Per-target error isolation
ensures one failed tweet or target never crashes the whole run.

***

### Local development

```bash
git clone https://github.com/Coderz-devz/twitter-scraper.git
cd twitter-scraper
python -m venv .venv && source .venv/Scripts/activate  # Windows Git Bash
pip install -r requirements.txt

## Set APIFY_MEMORY_STORAGE so the SDK uses local storage, not the cloud.
export APIFY_MEMORY_STORAGE=1
python -m src.main
```

#### Deploy to Apify

```bash
npm i -g apify          # install the Apify CLI
apify login             # paste your Apify API token
apify push              # builds the Docker image and deploys the Actor
```

***

#### Refreshing query ids

1. Open <https://x.com> (logged in) → DevTools → **Network** tab.
2. Filter by `TweetDetail`, open any tweet.
3. Right-click the `TweetDetail` request → **Copy URL**.
4. The URL looks like `.../graphql/<queryId>/TweetDetail?...`. Copy the id
   and update `QUERY_IDS["TweetDetail"]` in `src/scraper.py`.

***

### License

MIT — see [LICENSE](LICENSE).

# Actor input Schema

## `mode` (type: `string`):

v1 supports tweet detail/replies mode. Search and user-timeline modes are coming in a future release.

## `targets` (type: `array`):

One or more tweet URLs (https://x.com/user/status/123) or tweet IDs (123). The Actor fetches each tweet and (optionally) its reply thread.

## `maxTweets` (type: `integer`):

Maximum number of tweets to scrape per target URL/ID (caps the focal tweet + replies + related tweets returned in the same response).

## `includeReplies` (type: `boolean`):

If enabled, the Actor returns the focal tweet plus its reply thread. If disabled, only the focal tweet is kept.

## `authToken` (type: `string`):

Your X 'auth\_token' session cookie (REQUIRED). Get it from DevTools → Application → Cookies → x.com. See the README 'Getting your X cookies' section for step-by-step instructions.

## `ct0` (type: `string`):

Your X 'ct0' CSRF cookie (REQUIRED). Same location as auth\_token; its value is mirrored in the x-csrf-token header on every request.

## `requestDelay` (type: `number`):

Seconds to sleep between paginated requests (anti-ban politeness). Increase for large runs.

## `maxRetries` (type: `integer`):

Maximum retry attempts on HTTP 429 (rate limit) and 5xx (server error) responses, with exponential backoff.

## Actor input object example

```json
{
  "mode": "tweet",
  "targets": [
    "https://x.com/elonmusk/status/1784210000000000000"
  ],
  "maxTweets": 50,
  "includeReplies": true,
  "requestDelay": 1,
  "maxRetries": 5
}
```

# Actor output Schema

## `tweets` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "targets": [
        "https://x.com/elonmusk/status/1784210000000000000"
    ],
    "authToken": "",
    "ct0": ""
};

// Run the Actor and wait for it to finish
const run = await client.actor("api_heros/twitter-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "targets": ["https://x.com/elonmusk/status/1784210000000000000"],
    "authToken": "",
    "ct0": "",
}

# Run the Actor and wait for it to finish
run = client.actor("api_heros/twitter-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "targets": [
    "https://x.com/elonmusk/status/1784210000000000000"
  ],
  "authToken": "",
  "ct0": ""
}' |
apify call api_heros/twitter-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,api_heros/twitter-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/eegOEg7n9lfLSn6QR/builds/zoD13Pglh8ulvwfvc/openapi.json