xread: public X data from the command line, an API and an Apify Actor

One Go module that reads public X data: profiles, timelines, posts, threads, replies and search. It is a clean-room client of the GraphQL API that the x.com web app calls from every browser, the way FxTwitter reads X. Every read has a disk cache and a rate limit shared by every xread process on the machine, at most 1 request per second per X identity. The same reads run as a CLI (xread), as a loopback HTTP server behind the api's /v1/x route (xread serve), and as the Apify Actor x-twitter-scraper (cmd/xread-actor).

Rules it keeps

The captain's ruling of 2026-10-25 allows logins and dedicated scraper accounts for social platform data products. xread keeps the hard lines that stay:

• Public data only: no direct messages, no protected accounts or their posts, no email, phone, birth date or relationship fields. A protected account answers unavailable, and a protected author's post is dropped from a list.
• Signed-in reads use only a dedicated scraper account, from XREAD_X_AUTH_TOKEN and XREAD_X_CT0 (its auth_token and ct0 cookies) in the environment, never on the command line. Never the captain's own account, and never bought or borrowed credentials.
• At most 1 request per second per identity (the guest token, or the scraper account), enforced by the shared bucket: --rate above 1 is a usage error.
• No paid proxy, phone number or captcha service.

Build and run

cd apps/xread
go build -o bin/xread ./cmd/xread
bin/xread profile NASA @golang https://x.com/github
bin/xread timeline golang --pages 2
bin/xread post --full https://x.com/NASA/status/2113776971603689874
bin/xread thread https://x.com/github/status/2107896916595884177
bin/xread replies 2113776971603689874
bin/xread search "golang generics" --mode latest       # needs the scraper account
bin/xread post https://x.com/jack/status/20 --text     # the oEmbed reader, as before

bin/ is not committed; build it where you run it.

Commands

Command	Reads	Guest	Scraper account
profile <handle|url>...	1 to 20 accounts	yes	yes
timeline <handle|url>	posts and reposts, newest first, 20 a page	yes, every page	yes
post --full <url|id>...	1 to 20 posts with full long-post text, time, counts, views, links, media	yes	yes
thread <url|id>	the posts above it, the post, and the author's own follow-ups	first view only (partial: true)	full
replies <url|id>	the direct replies, 1 page at a time	first few only (partial: true)	every page
search <query>	X search, --mode latest or top	no: exit 3	yes
post <url|id>...	the oEmbed reader (publish.x.com/oembed), the read_x_post row shape	yes	not used
serve	the reads above over loopback HTTP	yes	yes

Flags: --cursor C starts a list from an earlier page's nextCursor; --pages N (1 to 10) reads several pages; --no-cache; --cache-dir DIR (else XREAD_CACHE_DIR, else the user cache directory plus /xread); --rate N (above 0, at most 1). The oEmbed post keeps its own flags (--text, --burst).

Exit codes: 0 every read worked, 1 at least one error row, 2 usage error, 3 the read needs the scraper account.

Output

profile and post --full print a JSON array of rows, in input order. A profile row: type: "profile", input, id, handle, name, url, description, location, website, createdAt (RFC 3339), followers, following, posts, verified, verifiedType, avatar, banner, pinnedPostIds. A post row: type: "post", input, id, url, text (t.co links expanded, the trailing media link dropped), createdAt, lang, author {id, handle, name, url, verified}, counts {replies, reposts, likes, quotes, bookmarks, views}, links, media [{type, url, preview, video, alt}], conversationId, replyToId, replyToHandle, quoteOfId, repostOf (the reposted post, on a repost), source.

timeline, thread, replies and search print one page object: type, input, user (timeline), focal (thread and replies), posts, nextCursor (null at the end), partial, note (what a partial read leaves out), auth (guest or account), source (x-graphql, or x-web-page for the logged-out page), and cached.

An error is the oEmbed reader's error row: type: "error", input, postId, code, message, retryable, status, and retryAfter on a rate limit. Codes: invalid_post (bad input), unavailable (not public, or does not exist), rate_limited, network_error, upstream_error, parse_error, not_available (needs the scraper account, or X rotated a query id).

How it reads X

• Guest: the public web-app bearer token plus a guest token from api.x.com/1.1/guest/activate.json, kept for 2 hours in <cache-dir>/guest-token.json so a shell loop does not ask for a new one per run. Guests can read UserByScreenName, UserTweets and TweetResultByRestId.
• The scraper account: its cookies, with ct0 as the CSRF header, on x.com/i/api/graphql; it adds TweetDetail (conversations) and SearchTimeline.
• Without the account, thread and replies read the logged-out post page, whose embedded data holds the post and its first few replies; the answer says partial: true.
• Query ids rotate when X ships its web app. Set XREAD_QUERY_IDS="TweetDetail=<id>,SearchTimeline=<id>" to swap one without a rebuild; the ids in internal/xweb/ops.go were read from x.com on 2026-10-25.
• The user agent is Mozilla/5.0 (compatible; agent-tools-xread/2.0; +https://tools.openkrill.app): it names the tool, and x.com serves the logged-out page to it.

Cache and rate limit

• Reads are cached on disk under <cache-dir>/x/<kind>/, keyed by the identity and the request: profiles and posts for 1 hour, conversations for 15 minutes, timeline and search pages for 5 minutes. A write goes to a temporary file and is renamed into place; a damaged file counts as a miss.
• Each identity has its own token bucket file, <cache-dir>/ratelimit-guest.json or ratelimit-account-<hash>.json, under an exclusive flock, so every process and every serve request share 1 request per second per identity.
• 429, 5xx and network errors are retried up to 3 tries with 1 s then 2 s backoff. On a 429 the scraper account holds every process until Retry-After or X's x-rate-limit-reset; a guest takes a new guest token instead, which comes with a fresh budget. A wait above 60 s ends the read with rate_limited and retryAfter.
• Retry log lines go to stderr and never include headers, cookies or tokens.

The API: xread serve and /v1/x

xread serve --addr 127.0.0.1:8879 answers GET /profile/{handle}, /timeline/{handle}?cursor=, /post/{id}, /thread/{id}, /replies/{id}?cursor=, /search?q=&mode=&cursor= and /health, with the CLI's JSON. It listens on loopback only; when XREAD_SERVE_TOKEN is set, each request must send it as a bearer token. It accepts GET and HEAD. Errors map to HTTP status codes: 400 bad input, 404 not public, 429 with Retry-After, 503 when the read needs the scraper account or the reader is busy pacing (with Retry-After), and 502 for an upstream fault, a rotated query id included. A 200 carries Cache-Control: private, max-age=60; errors carry no-store. A request gets 20 seconds and waits at most 10 seconds for a rate-limit hold, so it answers before the api's 25-second timeout.

On the VPS it runs as agent-tools-xread.service (ops/xread/, MemoryMax=128M), and the edge maps the api's X_READER binding to it. The api serves it at https://api.openkrill.app/v1/x/... behind an API key (x-api-key header); a call without a key gets 401 api_key_required.

curl -H "x-api-key: $KEY" https://api.openkrill.app/v1/x/timeline/golang

The Apify Actor

cmd/xread-actor and .actor/ make the Actor x-twitter-scraper: modes profile, timeline, post, thread and replies, one dataset item per profile or post, and one x-result event per item at 0.15 USD per 1,000 results (error items are free). It reads as a guest only: the scraper account's 1 request per second cap cannot be shared across Apify containers, so search and full conversations stay on /v1/x. The Dockerfile builds it; outside Apify, XREAD_ACTOR_INPUT='{"mode":"profile","handles":["NASA"]}' go run ./cmd/xread-actor prints the items as JSON lines. Publish with npm run apify:push -- xread --force (see docs/apify-publish.md). It is live on the Store since 2026-10-25 at https://apify.com/openkrill/x-twitter-scraper (Actor id lIuKTZh7yaq3jQ425, 256 MB).

Limits

• Guest conversations show only the first replies X gives a logged-out reader, and guests cannot search.
• A query id X rotates answers not_available with HTTP 404 until XREAD_QUERY_IDS or a rebuild names the new one.
• thread follows the author's follow-ups on the first page of the conversation only.

X search and trends without the account: Grok in an ego Space

Until the scraper account is set, X search and trend work can use Grok, which reads X live, in the OpenKrill Team account. The route is grok.com in a crew's own ego lite Space, driven by the ego-browser skill, never on the captain's screen:

1. Open a task Space, go to https://grok.com/, and check that the account menu says "OpenKrill Team".
2. Ask for author handle, post URL, date and a one-line summary, and say "only posts you found on X".
3. Save the answer and a screenshot in the task's data folder, then close the Space.
4. Check the posts you quote with xread post --full <url>....

Tests

cd apps/xread && go test ./...

These cover the oEmbed reader, the cache and the rate limit with a fake clock and a local server. The x.com reads are proved by live runs, saved with each task's report.