# X (Twitter) Scraper: Profiles, Posts, Threads, $0.15/1K (`openkrill/x-twitter-scraper`) Actor

Scrape public X (Twitter) profiles, user timelines, posts, threads and first replies into clean JSON: follower counts, full long-post text, likes, reposts, views and media. No login, no cookies, no fee per query. $0.15 per 1,000 results, and error items are free.

- **URL**: https://apify.com/openkrill/x-twitter-scraper.md
- **Developed by:** [Nguyen Vu Trung Hieu](https://apify.com/openkrill) (community)
- **Categories:** Social media, Lead generation, AI
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## xread: public X data from the command line, an API and an Apify Actor

One Go module that reads public X data: profiles, timelines, posts, threads, replies and search.
It is a clean-room client of the GraphQL API that the x.com web app calls from every browser, the way FxTwitter reads X.
Every read has a disk cache and a rate limit shared by every xread process on the machine, at most 1 request per second per X identity.
The same reads run as a CLI (`xread`), as a loopback HTTP server behind the api's `/v1/x` route (`xread serve`), and as the Apify Actor `x-twitter-scraper` (`cmd/xread-actor`).

### Rules it keeps

The captain's ruling of 2026-10-25 allows logins and dedicated scraper accounts for social platform data products.
xread keeps the hard lines that stay:

- Public data only: no direct messages, no protected accounts or their posts, no email, phone, birth date or relationship fields.
  A protected account answers `unavailable`, and a protected author's post is dropped from a list.
- Signed-in reads use only a dedicated scraper account, from `XREAD_X_AUTH_TOKEN` and `XREAD_X_CT0` (its `auth_token` and `ct0` cookies) in the environment, never on the command line.
  Never the captain's own account, and never bought or borrowed credentials.
- At most 1 request per second per identity (the guest token, or the scraper account), enforced by the shared bucket: `--rate` above 1 is a usage error.
- No paid proxy, phone number or captcha service.

### Build and run

```sh
cd apps/xread
go build -o bin/xread ./cmd/xread
bin/xread profile NASA @golang https://x.com/github
bin/xread timeline golang --pages 2
bin/xread post --full https://x.com/NASA/status/2113776971603689874
bin/xread thread https://x.com/github/status/2107896916595884177
bin/xread replies 2113776971603689874
bin/xread search "golang generics" --mode latest       # needs the scraper account
bin/xread post https://x.com/jack/status/20 --text     # the oEmbed reader, as before
```

`bin/` is not committed; build it where you run it.

### Commands

| Command | Reads | Guest | Scraper account |
| --- | --- | --- | --- |
| `profile <handle\|url>...` | 1 to 20 accounts | yes | yes |
| `timeline <handle\|url>` | posts and reposts, newest first, 20 a page | yes, every page | yes |
| `post --full <url\|id>...` | 1 to 20 posts with full long-post text, time, counts, views, links, media | yes | yes |
| `thread <url\|id>` | the posts above it, the post, and the author's own follow-ups | first view only (`partial: true`) | full |
| `replies <url\|id>` | the direct replies, 1 page at a time | first few only (`partial: true`) | every page |
| `search <query>` | X search, `--mode latest` or `top` | no: exit 3 | yes |
| `post <url\|id>...` | the oEmbed reader (`publish.x.com/oembed`), the `read_x_post` row shape | yes | not used |
| `serve` | the reads above over loopback HTTP | yes | yes |

Flags: `--cursor C` starts a list from an earlier page's `nextCursor`; `--pages N` (1 to 10) reads several pages; `--no-cache`; `--cache-dir DIR` (else `XREAD_CACHE_DIR`, else the user cache directory plus `/xread`); `--rate N` (above 0, at most 1).
The oEmbed `post` keeps its own flags (`--text`, `--burst`).

Exit codes: 0 every read worked, 1 at least one error row, 2 usage error, 3 the read needs the scraper account.

### Output

`profile` and `post --full` print a JSON array of rows, in input order.
A profile row: `type: "profile"`, `input`, `id`, `handle`, `name`, `url`, `description`, `location`, `website`, `createdAt` (RFC 3339), `followers`, `following`, `posts`, `verified`, `verifiedType`, `avatar`, `banner`, `pinnedPostIds`.
A post row: `type: "post"`, `input`, `id`, `url`, `text` (t.co links expanded, the trailing media link dropped), `createdAt`, `lang`, `author {id, handle, name, url, verified}`, `counts {replies, reposts, likes, quotes, bookmarks, views}`, `links`, `media [{type, url, preview, video, alt}]`, `conversationId`, `replyToId`, `replyToHandle`, `quoteOfId`, `repostOf` (the reposted post, on a repost), `source`.

`timeline`, `thread`, `replies` and `search` print one page object: `type`, `input`, `user` (timeline), `focal` (thread and replies), `posts`, `nextCursor` (null at the end), `partial`, `note` (what a partial read leaves out), `auth` (`guest` or `account`), `source` (`x-graphql`, or `x-web-page` for the logged-out page), and `cached`.

An error is the oEmbed reader's error row: `type: "error"`, `input`, `postId`, `code`, `message`, `retryable`, `status`, and `retryAfter` on a rate limit.
Codes: `invalid_post` (bad input), `unavailable` (not public, or does not exist), `rate_limited`, `network_error`, `upstream_error`, `parse_error`, `not_available` (needs the scraper account, or X rotated a query id).

### How it reads X

- Guest: the public web-app bearer token plus a guest token from `api.x.com/1.1/guest/activate.json`, kept for 2 hours in `<cache-dir>/guest-token.json` so a shell loop does not ask for a new one per run.
  Guests can read `UserByScreenName`, `UserTweets` and `TweetResultByRestId`.
- The scraper account: its cookies, with `ct0` as the CSRF header, on `x.com/i/api/graphql`; it adds `TweetDetail` (conversations) and `SearchTimeline`.
- Without the account, `thread` and `replies` read the logged-out post page, whose embedded data holds the post and its first few replies; the answer says `partial: true`.
- Query ids rotate when X ships its web app. Set `XREAD_QUERY_IDS="TweetDetail=<id>,SearchTimeline=<id>"` to swap one without a rebuild; the ids in `internal/xweb/ops.go` were read from x.com on 2026-10-25.
- The user agent is `Mozilla/5.0 (compatible; agent-tools-xread/2.0; +https://tools.openkrill.app)`: it names the tool, and x.com serves the logged-out page to it.

### Cache and rate limit

- Reads are cached on disk under `<cache-dir>/x/<kind>/`, keyed by the identity and the request: profiles and posts for 1 hour, conversations for 15 minutes, timeline and search pages for 5 minutes.
  A write goes to a temporary file and is renamed into place; a damaged file counts as a miss.
- Each identity has its own token bucket file, `<cache-dir>/ratelimit-guest.json` or `ratelimit-account-<hash>.json`, under an exclusive `flock`, so every process and every `serve` request share 1 request per second per identity.
- 429, 5xx and network errors are retried up to 3 tries with 1 s then 2 s backoff.
  On a 429 the scraper account holds every process until `Retry-After` or X's `x-rate-limit-reset`; a guest takes a new guest token instead, which comes with a fresh budget.
  A wait above 60 s ends the read with `rate_limited` and `retryAfter`.
- Retry log lines go to stderr and never include headers, cookies or tokens.

### The API: `xread serve` and `/v1/x`

`xread serve --addr 127.0.0.1:8879` answers `GET /profile/{handle}`, `/timeline/{handle}?cursor=`, `/post/{id}`, `/thread/{id}`, `/replies/{id}?cursor=`, `/search?q=&mode=&cursor=` and `/health`, with the CLI's JSON.
It listens on loopback only; when `XREAD_SERVE_TOKEN` is set, each request must send it as a bearer token.
It accepts GET and HEAD.
Errors map to HTTP status codes: 400 bad input, 404 not public, 429 with `Retry-After`, 503 when the read needs the scraper account or the reader is busy pacing (with `Retry-After`), and 502 for an upstream fault, a rotated query id included.
A 200 carries `Cache-Control: private, max-age=60`; errors carry `no-store`.
A request gets 20 seconds and waits at most 10 seconds for a rate-limit hold, so it answers before the api's 25-second timeout.

On the VPS it runs as `agent-tools-xread.service` (`ops/xread/`, `MemoryMax=128M`), and the edge maps the api's `X_READER` binding to it.
The api serves it at `https://api.openkrill.app/v1/x/...` behind an API key (`x-api-key` header); a call without a key gets 401 `api_key_required`.

```sh
curl -H "x-api-key: $KEY" https://api.openkrill.app/v1/x/timeline/golang
```

### The Apify Actor

`cmd/xread-actor` and `.actor/` make the Actor `x-twitter-scraper`: modes `profile`, `timeline`, `post`, `thread` and `replies`, one dataset item per profile or post, and one `x-result` event per item at 0.15 USD per 1,000 results (error items are free).
It reads as a guest only: the scraper account's 1 request per second cap cannot be shared across Apify containers, so search and full conversations stay on `/v1/x`.
The `Dockerfile` builds it; outside Apify, `XREAD_ACTOR_INPUT='{"mode":"profile","handles":["NASA"]}' go run ./cmd/xread-actor` prints the items as JSON lines.
Publish with `npm run apify:push -- xread --force` (see `docs/apify-publish.md`).
It is live on the Store since 2026-10-25 at https://apify.com/openkrill/x-twitter-scraper (Actor id `lIuKTZh7yaq3jQ425`, 256 MB).

### Limits

- Guest conversations show only the first replies X gives a logged-out reader, and guests cannot search.
- A query id X rotates answers `not_available` with HTTP 404 until `XREAD_QUERY_IDS` or a rebuild names the new one.
- `thread` follows the author's follow-ups on the first page of the conversation only.

### X search and trends without the account: Grok in an ego Space

Until the scraper account is set, X search and trend work can use Grok, which reads X live, in the OpenKrill Team account.
The route is grok.com in a crew's own ego lite Space, driven by the `ego-browser` skill, never on the captain's screen:

1. Open a task Space, go to `https://grok.com/`, and check that the account menu says "OpenKrill Team".
2. Ask for author handle, post URL, date and a one-line summary, and say "only posts you found on X".
3. Save the answer and a screenshot in the task's data folder, then close the Space.
4. Check the posts you quote with `xread post --full <url>...`.

### Tests

```sh
cd apps/xread && go test ./...
```

These cover the oEmbed reader, the cache and the rate limit with a fake clock and a local server.
The x.com reads are proved by live runs, saved with each task's report.

# Actor input Schema

## `mode` (type: `string`):

profile: one account per handle. timeline: the newest posts and reposts of each handle. post: each post by URL or id. thread: each post with the posts above it and the author's own follow-ups. replies: the first replies X shows a logged-out reader.

## `handles` (type: `array`):

For profile and timeline: handles such as NASA or @golang, or profile URLs such as https://x.com/github.

## `posts` (type: `array`):

For post, thread and replies: post URLs such as https://x.com/jack/status/20, or post ids.

## `maxItems` (type: `integer`):

The run stops after this many results in total.

## `maxPerInput` (type: `integer`):

For timeline: how many posts to read per handle (20 per page). Empty means the same as Max results.

## Actor input object example

```json
{
  "mode": "timeline",
  "handles": [
    "NASA",
    "golang",
    "github"
  ],
  "posts": [
    "https://x.com/jack/status/20"
  ],
  "maxItems": 20
}
```

# Actor output Schema

## `posts` (type: `string`):

No description

## `profiles` (type: `string`):

No description

## `dataset` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "handles": [
        "NASA",
        "golang",
        "github"
    ],
    "posts": [
        "https://x.com/jack/status/20"
    ],
    "maxItems": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("openkrill/x-twitter-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "handles": [
        "NASA",
        "golang",
        "github",
    ],
    "posts": ["https://x.com/jack/status/20"],
    "maxItems": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("openkrill/x-twitter-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "handles": [
    "NASA",
    "golang",
    "github"
  ],
  "posts": [
    "https://x.com/jack/status/20"
  ],
  "maxItems": 20
}' |
apify call openkrill/x-twitter-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,openkrill/x-twitter-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/lIuKTZh7yaq3jQ425/builds/CGEd5JQdzv6FYmdsq/openapi.json