# X Tweet Scraper (`devchrisvaz/x-tweet-scraper`) Actor

- **URL**: https://apify.com/devchrisvaz/x-tweet-scraper.md
- **Developed by:** [Alexander Vazquez](https://apify.com/devchrisvaz) (community)
- **Categories:** Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## X Tweet Scraper for Apify

This Actor retrieves public X author timelines and individual tweets over X's HTTP GraphQL endpoints, normalizes them into a stable dataset contract, and enforces the entitlement decision before any tweet is emitted. It is designed for authorized assessment and research workloads only. It does not use a browser, automate login, evade an access wall, or claim that an X response is available when it is not.

### Architecture and data flow

The Apify Actor receives validated input, obtains its platform identity from Actor.getEnv(), resolves the run's entitlement with the signer, discovers the current public X operation manifest, keeps one guest session and one configured proxy transport, fetches an author profile/timeline or tweet by ID, normalizes and filters candidate tweets, reserves tweet IDs, and emits only IDs covered by a signed reservation. The coordinator checkpoints cursors, seen IDs, pending grants, and run statistics in Apify state and writes one output metadata record at completion.

The Vercel Node.js 24 Functions are the entitlement authority. They authenticate Actor requests with one canonical JSON body plus timestamp/nonce headers and an HMAC, freeze the first decision for (actorId, runId, userId), sign resolutions and reservation decisions with Ed25519, and reserve SHA-256 tweet-ID hashes atomically in the Marketplace-provisioned Upstash Redis instance. Reservation records have a seven-day TTL and retries are idempotent. Redis state is authoritative; local Actor state is a resumable checkpoint, never a billing authority.

```text
Apify input -> Actor.getEnv identity -> HMAC signer request -> Vercel Functions
      ^                                                        |
      |                                                        v
dataset <- emission guard <- signed grant <- Redis Lua reservation <- Ed25519 signer
```

### Input and output

The supported input is defined by INPUT\_SCHEMA.json. Use a non-empty fromUsers array for author timelines or a non-empty tweetIds array for individual tweets. maxResults, date/language/engagement filters, reply/retweet flags, media filters, verification, and the documented Apify proxy configuration are supported. searchTerms is rejected by the platform schema and runtime: this Actor does not pretend to implement search through X's top authentication wall.

#### Authentication (Bypassing Guest Limits)

X (Twitter) heavily rate-limits or blocks standard GraphQL Guest Sessions, often returning `422 Unprocessable Entity` or `403 Forbidden` errors. To bypass these new anti-scraping blocks, you can provide an `authCookies` array in the input (e.g., `["auth_token=...; ct0=..."]`).

When `authCookies` are provided, the scraper operates as an authenticated session. It automatically extracts the `ct0` value from the provided cookie string to generate the required `x-csrf-token` header.

**⚠️ WARNING:** Scraping as an authenticated user carries a high risk of account suspension. It is strongly recommended to use **throwaway accounts** when utilizing the `authCookies` feature. Do not use your primary or valuable X accounts.

Each dataset item follows OUTPUT\_SCHEMA.json, including stable author, metrics, entities, media, source, UTC createdAt, and UTC-Z scrapedAt. Run-level tier, effective limit, and discovered, filtered, reserved, emitted, denied, and errors statistics are written to the persisted output metadata contract. Malformed X graph shapes are rejected rather than turned into partial output.

Supported HTTP-only surfaces are:

- author profile lookup and the author's tweet timeline;
- individual tweet lookup by REST ID;
- the profile lookup used to resolve an author ID before timeline paging.

Quote/reply/retweet metadata may be normalized when present in a returned tweet, but a separate search endpoint is not promised. X operation IDs are discovered from the public manifest and verified bootstrap IDs are retained as a fallback. A build-key change invalidates the cache; operation drift receives one discovery refresh and then fails closed.

### Authentication and Anti-Scraping Bypasses

X (Twitter) heavily throttles and restricts standard Guest Sessions, frequently blocking requests with `422 Unprocessable Entity`, `401 Unauthorized`, or `403 Forbidden` errors. The platform increasingly requires valid, logged-in session cookies to successfully resolve GraphQL endpoints such as `UserTweets` and `UserByScreenName`.

To bypass these blocks, this scraper supports an **authenticated session mode** using the optional `authCookies` input.

#### Configuring `authCookies`

If provided in your Apify input, `authCookies` must be an array of strings representing your session cookies (specifically, you must provide your `auth_token` and `ct0` cookies). The scraper's session manager will use these cookies to dynamically mimic an authenticated user, automatically refresh the required `x-csrf-token` (the `ct0` cookie) when necessary, and adapt the GraphQL payloads to the authenticated JSON shape.

**Example Input configuration:**

```json
{
  "authCookies": [
    "auth_token=d5ff4a...; ct0=e78c4b..."
  ]
}
```

**⚠️ IMPORTANT WARNING:**
Scraping X while logged into an account poses a significant risk to that account. X actively monitors for automated behavior, and using your primary personal account for scraping **will likely result in a permanent ban**. You must **ONLY** use throwaway/burner accounts for the `authCookies` input.

### Entitlement and security model

Only Apify's platform-controlled Actor.getEnv() fields are trusted for identity and payer status: actorId, actorRunId, userId, and the documented userIsPaying value "1". Input fields, user environment, and a caller-provided override cannot assert a paid tier. The Vercel Function never reads a deployment-global payer bit. Missing identity, an unrecognized payer value, signer outage, invalid HMAC, replay, actor mismatch, invalid Ed25519 signature, expired grant, or Redis mutation failure freezes the run to free/unknown and prevents unreserved emission.

The first resolved limit for a run is sticky: free is capped at 10, paid is capped at the requested maximum, and a later request can never raise a frozen limit. The Actor verifies the subject, key ID, expiry, limit, and Ed25519 signature before accepting a resolution or reservation. The Lua reservation is the only concurrent cap authority. It grants each new hash at most once, grants duplicate retries idempotently, denies after the frozen cap, and preserves the seven-day TTL. Tweet IDs are not stored in Redis; only their SHA-256 hashes are stored. Subject keys are hashes of canonical {actorId,runId,userId} JSON, so delimiters or user-controlled characters cannot collide.

Forked or copied Actor runs do not inherit a grant: the signer binds every request and signature to the exact actor, run, and user subject. A copied local state file cannot grant output without a fresh signer decision. Sticky proxies preserve one transport identity for a run; proxy rotation is not used to evade limits, access controls, or rate limits.

### State and migration

Apify state is versioned and contains target cursors, visited cursors, seen IDs, pending tweets, and run statistics. Restored pending tweets are reconciled before new work. The entitlement repository accepts the current persisted run representation and migrates legacy grantedHashes arrays to the current granted map during initialization. Redis KEEPTTL is used for atomic reservations so a mutation cannot silently extend a grant. A state schema change must increment the version and add a migration before deployment; never hand-edit a production Redis key.

### Local checks and smoke commands

Use Node.js 24 (the repository pins >=24 <25 in package.json and .node-version, enforces engine-strict in .npmrc, and runs a preflight check):

```sh
npm ci
npm run preflight:node
npm run verify
npm run fixture:run
npm run signer:smoke
```

fixture:run uses deterministic in-memory X and entitlement fixtures and never contacts X. signer:smoke generates an ephemeral Ed25519 key, verifies a paid requested cap, and exercises duplicate reservations without printing key material. check:browser fails if a browser engine dependency enters the lockfile. validate:packaging checks the Dockerfile, Apify manifest, Vercel runtime, ESLint ignores, and that policy.

The live commands are opt-in and HTTP-only. They require explicit targets and confirmation, for example LIVE\_SMOKE\_CONFIRM=1 LIVE\_X\_USERNAME=... LIVE\_X\_TWEET\_ID=... npm run smoke:live; an optional LIVE\_X\_PROXY\_URL must be a permitted sticky proxy. They use real profile, author-timeline, and tweet responses and fail if a response is missing an ID; they never fabricate results. benchmark:live has the same explicit-target requirement and reports the honest status UNMEASURED\_UNTIL\_TASK6\_PAID\_RUN. No live smoke, paid behavior, or benchmark result is claimed by this repository.

### Deployment

The production Actor is built by Dockerfile from Node 24 and starts dist/index.js. .actor/actor.json points to that Dockerfile and the checked-in input/dataset schemas. The Vercel Functions in api/entitlements/ use the supported default Node.js runtime (with a 300-second limit) in vercel.json; its empty build command skips the root Actor build, while the explicit `public` output contains only an inert `robots.txt` and cannot expose source files. package.json pins deployments to Node 24 and they are not Edge Functions.

The intended/provisioned names are Vercel project x-tweet-scraper-entitlements and Marketplace resource upstash-kv-cordovan-village. Connection of that resource, secret configuration, Vercel deployment, and Apify deployment remain Task 6 actions; this worktree makes no deployment claim. The Marketplace normally injects KV\_REST\_API\_URL and KV\_REST\_API\_TOKEN. The signer accepts those names and falls back to UPSTASH\_REDIS\_REST\_URL and UPSTASH\_REDIS\_REST\_TOKEN; when both mappings exist, the UPSTASH\_REDIS\_REST\_\* aliases take precedence. The example contains names and harmless placeholders only; private keys, HMACs, REST tokens, and .env.local values must never be committed or printed. The Actor receives only its endpoint, HMAC, and pinned public key through the deployment secret mechanism; payer status continues to come from Apify runtime identity.

Task 6 operator checklist (authenticated operator only; not executed here):

1. In the Vercel dashboard, confirm project x-tweet-scraper-entitlements and attach the existing resource upstash-kv-cordovan-village; do not create a second Redis resource.
2. Link locally with `vercel link --project x-tweet-scraper-entitlements`, pull only to an ignored file with `vercel env pull .env.local`, and confirm the attached resource supplies KV\_REST\_API\_URL/KV\_REST\_API\_TOKEN (or explicitly configure the UPSTASH\_REDIS\_REST\_\* aliases). Do not place secret values in commands, logs, or commits.
3. Run `npm ci`, `npm run verify`, and `npm run docker:build` where Docker is available; inspect the generated deployment configuration.
4. After reviewing the pinned public key and canonical actor binding, an authorized operator may run `vercel deploy --prod` and the normal Apify Actor release command (`apify push`). These commands were not run for Task 5.

CI runs clean npm ci, browser-engine policy, packaging validation, lint, strict typecheck, the complete Vitest suite, build, and a Docker build on Node 24; both CI jobs that invoke npm use setup-node with Node 24. The workflow does not publish or deploy. A deployment gate should run these checks again and verify the configured signer public key and actor binding before accepting traffic.

### Limitations, cost, and authorization

X can change public manifests, operation IDs, guest access, response shapes, rate limits, or terms without notice. Public guest access can fail even when the code is healthy. Apify compute, dataset, proxy, Vercel Function, and Upstash request/storage costs are workload-dependent; this repository does not claim a cost benchmark. Search, authenticated/private content, login, browser automation, CAPTCHA solving, and access-wall evasion are out of scope.

Use is limited to an unlisted assessment or other workload for which the operator has authorization from the data owner and the relevant platform terms permit the requested collection. Review current X terms, developer policy, privacy obligations, retention requirements, and applicable law before operating the Actor. A successful HTTP response is not permission to collect or redistribute content.

# Actor input Schema

## `fromUsers` (type: `array`):

Array of X usernames to scrape.

## `tweetIds` (type: `array`):

Array of specific Tweet IDs to scrape.

## `searchTerms` (type: `array`):

Unsupported.

## `hashtags` (type: `array`):

Array of hashtags to scrape.

## `since` (type: `string`):

ISO-8601 date or timestamp, inclusive.

## `until` (type: `string`):

ISO-8601 date or timestamp, inclusive.

## `language` (type: `string`):

Language code of the tweets.

## `minLikes` (type: `integer`):

Minimum number of likes.

## `minRetweets` (type: `integer`):

Minimum number of retweets.

## `minReplies` (type: `integer`):

Minimum number of replies.

## `onlyVerified` (type: `boolean`):

Only scrape tweets from verified users.

## `mediaType` (type: `string`):

Filter tweets by media type.

## `includeReplies` (type: `boolean`):

Include replies in the scraped tweets.

## `includeRetweets` (type: `boolean`):

Include retweets in the scraped tweets.

## `sortBy` (type: `string`):

Sort order of the tweets.

## `maxResults` (type: `integer`):

Maximum number of results to return.

## `proxyConfiguration` (type: `object`):

Proxy configuration to use.

## `authCookies` (type: `array`):

Array of X authentication cookies (e.g. 'auth\_token=...; ct0=...'). If provided, the scraper will use them instead of Guest Tokens.

## Actor input object example

```json
{
  "fromUsers": [],
  "tweetIds": [],
  "searchTerms": [],
  "hashtags": [],
  "onlyVerified": false,
  "mediaType": "any",
  "includeReplies": true,
  "includeRetweets": false,
  "sortBy": "latest",
  "maxResults": 100,
  "authCookies": []
}
```

# Actor output Schema

## `id` (type: `string`):

Unique identifier of the tweet

## `url` (type: `string`):

URL of the tweet

## `text` (type: `string`):

Content of the tweet

## `createdAt` (type: `string`):

UTC timestamp of the tweet

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("devchrisvaz/x-tweet-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("devchrisvaz/x-tweet-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call devchrisvaz/x-tweet-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,devchrisvaz/x-tweet-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/F0yQGnAJ06UQKpV0e/builds/YWVlcoMeHCXqaBN46/openapi.json