# Twitter Post Scraper | No Cookie | No Login (`rexreus/twitter-post-scraper`) Actor

Scrape public X (Twitter) tweets, profiles & timelines at scale — no login needed. Filter by date, replies & retweets; export clean JSON/CSV/Excel. Duplicate-free resume, cost controls & self-healing when X changes its API. Fast, reliable, crash-proof.

- **URL**: https://apify.com/rexreus/twitter-post-scraper.md
- **Developed by:** [REXREUS D.O](https://apify.com/rexreus) (community)
- **Categories:** Lead generation, SEO tools, Social media
- **Stats:** 1 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

from $0.30 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

<div align="center">

## 🐦 Twitter (X) Post Scraper

<img src="https://i.imgur.com/EQ3vJa5.jpeg" align="center">

**Fast, cost-efficient, and crash-proof scraper for public X.com tweets, profiles & timelines.**

**Hybrid/Tiered** architecture • **Public data only** (guest-token) • **Never crashes** thanks to three runtime guards.

![Platform](https://img.shields.io/badge/platform-Apify_Actor-2b3a55)
![Runtime](https://img.shields.io/badge/node-%E2%89%A520-3c873a)
![Language](https://img.shields.io/badge/TypeScript-ESM-3178c6)
![Scope](https://img.shields.io/badge/scope-public--only-0a7)
![Tests](https://img.shields.io/badge/tests-200_passing-2ea44f)

</div>

***

### ✨ Why this scraper?

| | |
| --- | --- |
| ⚡ **Fast & cheap** | Tier 1 uses direct HTTP with TLS-impersonation — high volume, minimal proxy bytes. |
| 🛡️ **Crash-proof** | Three runtime guards always terminate **cleanly** with a specific `terminationReason` — it never falls over. |
| 🔁 **Self-healing** | Automatically re-harvests `queryId`/`features` & regenerates `x-client-transaction-id` when X changes its API. |
| ♻️ **Duplicate-free resume** | State & cursors are saved to the Key-Value Store; resumed runs never write the same tweet twice. |
| 💰 **Cost-aware** | A Cost Circuit Breaker stops the run before proxy/dataset charges spiral out of control. |
| 🔒 **Safe & compliant** | Read-only, public data only, and never logs tokens or secrets. |

***

### 🚀 Quick Start

1. Provide **at least one** of `handles` **or** `startUrls`.
2. Use a **RESIDENTIAL** proxy (required — datacenter IPs are permanently banned by X.com).
3. Run the Actor. Scraped tweets land in the **Apify Dataset**.

```json
{
    "handles": ["nasa", "spacex"],
    "maxTweetsPerUser": 100,
    "includeReplies": false,
    "includeRetweets": true,
    "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }
}
```

> 💡 **Use cases:** brand sentiment analysis, competitor research, keyword monitoring,
> academic research datasets, or archiving public account timelines.

***

### 🧭 How it works

```
INPUT ─▶ Fail-fast validation ─▶ Tier Router ─┬─▶ Tier 1: HTTP-direct (TLS-impersonation)   ◀── primary
        (handles/startUrls)                    └─▶ Tier 2: Playwright + CDP interception      ◀── fallback
                                                       │
                                    Dedup + Cursor + Enrichment (optional)
                                                       │
                                    Apify Dataset  +  Key-Value Store (STATE)
```

1. **Fail-fast input validation** — at least one of `startUrls`/`handles` is required, checked
   **before** any proxy or resource is used.
2. **Tier router** — `auto` (Hybrid), or force `http` / `browser`.
3. **Per-target scrape** — page-by-page with cursors, dedup, and state checkpoints.
4. **Three runtime guards** ensure the Actor **always terminates cleanly** (see [Resilience](#-resilience)).

#### 🔀 Two transport tiers

| | **Tier 1 — HTTP-direct** | **Tier 2 — Browser-CDP** |
| --- | --- | --- |
| Engine | `got-scraping` + TLS-impersonation | Playwright + CDP request interception |
| Role | **Primary** (default) | **Fallback** / protected operations |
| Proxy cost | Low | **10–50× higher** |
| Used when | Normal volume | Self-heal fails / JS execution needed |

> **Locked scope — public data only.** No DMs, protected accounts, follower lists, or full Search.
> Read-only (no write actions). The authenticated path is an opt-in input only
> (`authToken`/`csrfToken`, secret) and is **disabled by default** — using it is at the user's own
> ToS risk.

***

### ⚙️ Input

Provide **at least one** of `startUrls` or `handles`.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `startUrls` | array | – | X.com profile or single-tweet URLs. |
| `handles` | array | – | X.com usernames without the leading `@`. |
| `mode` | `auto`|`http`|`browser` | `auto` | Force a transport tier or let the router decide. |
| `maxTweetsPerUser` | integer (≥1) | `100` | Max tweets per target. Ignored when `noPerUserLimit` is on. |
| `noPerUserLimit` | boolean | `false` | Collect all tweets per target (still bounded by `maxRequestsPerCrawl`). |
| `maxRequestsPerCrawl` | integer (≥1) | `5000` | Hard guard against runaway cost & memory. |
| `proxyByteBudgetMb` | integer (≥1) | – | Cost breaker: stop the run cleanly once proxy traffic exceeds the budget. |
| `includeReplies` | boolean | `false` | Include reply tweets. |
| `includeRetweets` | boolean | `true` | Include retweets. |
| `dateFrom` / `dateTo` | string | – | Date filter (ISO 8601 / `YYYY-MM-DD`), derived from the Snowflake ID. |
| `enableEnrichment` | boolean | `false` | Enable optional AI enrichment (non-blocking). |
| `enrichment` | object | – | Enrichment options, e.g. `{ "sentiment": true, "translateTo": "id" }`. |
| `proxyConfiguration` | object | RESIDENTIAL | Apify proxy configuration (residential required). |
| `authToken` / `csrfToken` | string (secret) | – | Optional authenticated mode. Disabled by default. |

> **No sentinel values.** "Unlimited" is the boolean `noPerUserLimit`, **not**
> `maxTweetsPerUser: 0` (the minimum stays `1`).

<details>
<summary><b>📋 Full input example</b></summary>

```json
{
    "handles": ["nasa", "spacex"],
    "startUrls": [{ "url": "https://x.com/nasa" }],
    "mode": "auto",
    "maxTweetsPerUser": 100,
    "noPerUserLimit": false,
    "maxRequestsPerCrawl": 5000,
    "proxyByteBudgetMb": 512,
    "includeReplies": false,
    "includeRetweets": true,
    "dateFrom": "2025-01-01",
    "enableEnrichment": false,
    "proxyConfiguration": {
        "useApifyProxy": true,
        "apifyProxyGroups": ["RESIDENTIAL"]
    }
}
```

</details>

***

### 📦 Output (Apify Dataset)

Each Dataset item is a single **flat tweet record** (<9 MB, media stored as URLs only).

```json
{
    "tweetId": "1234567890123456789",
    "url": "https://x.com/exampleuser/status/1234567890123456789",
    "text": "This is an example tweet body.",
    "createdAt": "2025-01-15T12:00:00.000Z",
    "lang": "en",
    "author": {
        "id": "44196397",
        "handle": "exampleuser",
        "name": "Example User",
        "verified": true,
        "followersCount": 15000
    },
    "metrics": {
        "replyCount": 12,
        "retweetCount": 34,
        "likeCount": 560,
        "quoteCount": 7,
        "bookmarkCount": 21,
        "viewCount": 98000
    },
    "media": ["https://pbs.twimg.com/media/EXAMPLE1.jpg"],
    "isReply": false,
    "isRetweet": false,
    "isQuote": false,
    "conversationId": "1234567890123456789",
    "enrichment": null,
    "scrapedAt": "2025-01-15T12:05:00.000Z",
    "sourceTier": "http"
}
```

- `createdAt` is derived from the **Snowflake ID**: `(id >> 22) + 1288834974657`.
- `sourceTier` marks the tier (`http`/`browser`) that produced the record.
- `enrichment` is `null` when enrichment is disabled/skipped.

#### 🗃️ Key-Value Store (State & Cache)

The Actor persists operational state for **duplicate-free resume** after `persistState`/`migrating` events.

| Key | Contents | Purpose |
| --- | --- | --- |
| `STATE` | Per-target cursors, dedup index, and request budget. | Persisted for safe resume; holds `terminationReason` when the run stops. |
| `CACHE_QUERYIDS` | GraphQL `queryId` + `features` toggles (TTL'd). | Reused until expiry to reduce re-harvesting. |
| `TOKEN_POOL` | Guest-token ↔ proxy-IP affinity (ephemeral, rotated). | **No user credentials.** |

***

### 🛡️ Resilience

Three independent guards; all terminate **cleanly** (state flushed, resume-able, with a specific `terminationReason`):

1. **💰 Cost Circuit Breaker** (`cost-governor.ts`) — input `proxyByteBudgetMb` + env
   `ACTOR_MAX_TOTAL_CHARGE_USD` / `ACTOR_MAX_PAID_DATASET_ITEMS`. SOFT 80% = stop new targets,
   force Tier 2→1; HARD 100% = clean exit `cost_budget_exceeded`. Has absolute authority.
2. **🔌 Error-Rate Circuit Breaker** (`circuit-breaker.ts`) — `CLOSED→OPEN→HALF_OPEN`, sliding
   window of 20 per tier, counting **auth-class errors only** (401/403/`tid_rejected`/
   `queryId_unknown`; 429 goes to proxy quarantine). OPEN triggers self-heal then Tier 2 failover;
   persistent failure → `tid_algorithm_drift`.
3. **📉 Degradation State Machine** (`orchestrator.ts`) — `HEALTHY→DEGRADED→ABORTED`. Global
   guest-access revocation → DEGRADED → clean exit `guest_access_revoked` (or stay alive if the
   user supplied an `authToken` escape-hatch).

| `terminationReason` | Meaning |
| --- | --- |
| `completed` | Finished normally. |
| `cost_budget_exceeded` | Cost breaker hit HARD 100%. |
| `tid_algorithm_drift` | Error-rate breaker OPEN & Tier 2 failover exhausted. |
| `guest_access_revoked` | Global guest access revoked, no auth escape-hatch. |
| `max_requests_reached` | `maxRequestsPerCrawl` limit reached. |

***

### ❓ FAQ & Troubleshooting

<details>
<summary><b><code>0 tweets</code> with a <code>"The Proxy external access feature isn't enabled"</code> log</b></summary>

X.com **permanently blocks datacenter IPs**, so the Actor needs an **Apify RESIDENTIAL proxy**.
Make sure your Apify account has active residential proxy access and that `proxyConfiguration`
uses the `RESIDENTIAL` group. Without a residential proxy, requests are very likely rejected.

</details>

<details>
<summary><b>Running without a proxy (<code>useApifyProxy: false</code>)</b></summary>

Allowed (direct connection), but the Actor logs a **warning** because X.com typically rejects
datacenter IPs. This suits local testing only, **not** production. For reliable results, always
use a residential proxy.

</details>

<details>
<summary><b>How do I collect <i>all</i> tweets from an account?</b></summary>

Enable `noPerUserLimit: true`. The total is still bounded by `maxRequestsPerCrawl` as a
cost/memory guard — raise it if needed.

</details>

<details>
<summary><b>Run stopped with <code>cost_budget_exceeded</code></b></summary>

The cost budget was reached. Raise `proxyByteBudgetMb` or the `ACTOR_MAX_*` env vars if intended,
then **resume** the run — the Actor continues from the saved cursor without duplicates.

</details>

***

### 📈 Pricing & Support

This Actor is a commercial product available on the Apify Store. Pricing follows your Apify plan
and the proxy/compute resources consumed per run. For questions, feature requests, or custom
scraping needs, contact us through the Actor's Apify Store page.

> **Compliance:** Use for **public data only** and comply with X.com's Terms of Service and
> applicable law. This Actor is read-only and performs no write actions.

# Actor input Schema

## `startUrls` (type: `array`):

X.com profile or single-tweet URLs to scrape.

## `handles` (type: `array`):

X.com usernames without the leading @.

## `mode` (type: `string`):

Force a transport tier or let the router decide.

## `maxTweetsPerUser` (type: `integer`):

Maximum tweets to collect per target. Ignored when 'No per-user limit' is enabled.

## `noPerUserLimit` (type: `boolean`):

Collect all available tweets per target, ignoring 'Max tweets per user' (still bounded by 'Max requests per crawl').

## `maxRequestsPerCrawl` (type: `integer`):

Hard guard against runaway cost and memory usage.

## `proxyByteBudgetMb` (type: `integer`):

Cost circuit breaker: stop the run cleanly once residential-proxy traffic exceeds this budget. Tier 2 (Browser) uses 10-50x more bandwidth than Tier 1, so this guards against runaway cost (finding D-1). Leave empty to rely only on platform Max total charge / Max items.

## `includeReplies` (type: `boolean`):

Include reply tweets in a user's timeline.

## `includeRetweets` (type: `boolean`):

Include retweets in a user's timeline.

## `dateFrom` (type: `string`):

Only include tweets on or after this date (derived from Snowflake timestamp). ISO 8601 or YYYY-MM-DD.

## `dateTo` (type: `string`):

Only include tweets on or before this date. ISO 8601 or YYYY-MM-DD.

## `enableEnrichment` (type: `boolean`):

Enable optional, non-blocking AI enrichment (sentiment / translation).

## `enrichment` (type: `object`):

Configuration for AI enrichment. Only used when enableEnrichment is true.

## `proxyConfiguration` (type: `object`):

Residential proxy is required — datacenter IPs are permanently banned by X.com.

## `authToken` (type: `string`):

OPTIONAL authenticated-mode token. Leave empty for public guest-only scraping. Using this enables login-scoped access at the user's own ToS risk.

## `csrfToken` (type: `string`):

OPTIONAL CSRF (ct0) token, only used together with Auth token.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://x.com/nasa"
    }
  ],
  "handles": [
    "nasa",
    "spacex"
  ],
  "mode": "auto",
  "maxTweetsPerUser": 100,
  "noPerUserLimit": false,
  "maxRequestsPerCrawl": 5000,
  "includeReplies": false,
  "includeRetweets": true,
  "enableEnrichment": false,
  "enrichment": {
    "sentiment": true,
    "translateTo": "id"
  },
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

# Actor output Schema

## `dataset` (type: `string`):

Dataset containing all scraped, normalized tweet records

## `state` (type: `string`):

Key-value store containing run state, query ID cache, and token pool

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://x.com/nasa"
        }
    ],
    "handles": [
        "nasa",
        "spacex"
    ],
    "enrichment": {
        "sentiment": true,
        "translateTo": "id"
    },
    "proxyConfiguration": {
        "useApifyProxy": true,
        "apifyProxyGroups": [
            "RESIDENTIAL"
        ]
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("rexreus/twitter-post-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://x.com/nasa" }],
    "handles": [
        "nasa",
        "spacex",
    ],
    "enrichment": {
        "sentiment": True,
        "translateTo": "id",
    },
    "proxyConfiguration": {
        "useApifyProxy": True,
        "apifyProxyGroups": ["RESIDENTIAL"],
    },
}

# Run the Actor and wait for it to finish
run = client.actor("rexreus/twitter-post-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://x.com/nasa"
    }
  ],
  "handles": [
    "nasa",
    "spacex"
  ],
  "enrichment": {
    "sentiment": true,
    "translateTo": "id"
  },
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}' |
apify call rexreus/twitter-post-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,rexreus/twitter-post-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/GdEdjN1TXc4CfTldr/builds/tvAg38On7g9eLg3MQ/openapi.json