X / Twitter Scraper — Posts, Profiles & Monitoring

Scrape public X/Twitter profiles, posts and paginated profile timelines without an X login. Capture visible replies from public post pages and monitor the same targets for new posts and public metric changes. No X cookies or account setup required.

Quick start

{ "targets": ["@NASA"] }

The default collects up to 25 posts per profile, plus the profile record. You can mix public handles, profile URLs and post URLs:

{
  "targets": [
    "@NASA",
    "https://x.com/OpenAI",
    "https://x.com/NASA/status/2102882545427714266"
  ],
  "maxPosts": 100,
  "maxPages": 6,
  "scrapeReplies": true
}

Supported targets are a valid handle (@NASA or NASA), an X/Twitter profile URL, a post URL, or explicit profile: and post: prefixes. Search terms, hashtags, reply/media/repost timeline prefixes, follower/following targets and Lists are rejected with a clear error. Media, quotes and repost relationships can still appear in posts discovered through supported targets.

Ready-made Tasks

The release includes five reusable Apify Tasks built only around surfaces verified without an X account:

• Profile & Recent Timeline — profile + bounded public timeline pagination
• Post & Visible Replies — one public post plus replies exposed by the logged-out page
• Profile Monitor — new posts and public metric/profile changes across scheduled runs
• Multi-Profile Research — scrape several public profiles in one deduplicated run
• High-Engagement Posts — filter a public profile timeline by engagement thresholds

No Task asks for X cookies, credentials, OAuth, or account login. Search/followers/Lists Tasks are intentionally not offered because those surfaces are not currently reliable logged-out.

What comes back

• Profile records: stable user ID when exposed, handle, display name, biography, public counts, verification flags, images and other available metadata.
• Post records: canonical URL, text, timestamp, author, public engagement counts, media URLs, entities, and reply/quote/repost relationships where exposed.
• Change records in monitoring mode: previous and current metrics for a post or tracked profile.

Missing upstream fields remain null; missing counts are not invented as zero. Records include recordType, scrapedAt and schemaVersion. The same post found through multiple supported targets is deduplicated by X ID and retains provenance. Common post fields (authorUsername, likes, replies, reposts, views, mediaCount) are also flattened for CSV/Excel; nested author, engagement and media objects remain in JSON.

For post URL targets, scrapeReplies: true adds connected replies present in the public page payload. It is not a complete conversation. The tested post showed a public count of 93 replies while only three connected replies were accessible in that first window. maxReplies, includeNestedReplies and maxReplyDepth bound traversal of relationships actually present.

Profile timelines begin with public HTML and continue through guest GraphQL cursors when available. maxPosts, maxPages, dateFrom and dateTo bound collection. Cursor loops, duplicate-only pages and continuation failures stop safely; already collected valid records are retained and the target is marked partial in RUN_SUMMARY.

Monitoring

{
  "targets": ["@NASA"],
  "monitor": true,
  "stateKey": "nasa-x",
  "emitMode": "new_and_changed",
  "maxResults": 100
}

emitMode is all, new, changed or new_and_changed. The Actor stores bounded ID and metric snapshots in an Apify Key-Value Store. maxStateRecords and changeRetentionDays prune history. Records excluded by the delivery cap or charge limit are not marked as delivered. Use the same stateKey in scheduled runs to continue monitoring.

Three separate private Apify runs with one state key verified all → new → new_and_changed: the first two capped runs emitted ten distinct records each and stored ten then twenty snapshots. The third emitted real metric/profile changes and one previously unseen post. Another stable-post new run emitted zero rows cleanly.

Filters and limits

Local filters include dates, minimum engagement counts, author inclusion/exclusion, keywords, languages, media type, verified authors, replies/roots, and quote/repost inclusion. Sort by time or engagement metrics. Filters run after normalization and deduplication. RUN_SUMMARY reports filtered and droppedByResultCap separately.

The default is direct HTTP. Guest tokens are acquired automatically, reused and rotated when needed. An optional Apify proxy configuration is available for retries; no X account credentials or cookies are accepted. Residential escalation and live rate-limit recovery are not yet proven.

Dataset and billing

One mixed dataset contains profile, post and change records. Dataset views named “Post columns”, “Profile columns”, “Reply columns” and “Change columns” are column projections, not record-type filters. Filter on recordType in an export or integration; replies are posts with isReply: true. Raw nested JSON remains in the dataset.

Launch pay-per-event pricing is $0.40 per 1,000 delivered posts/visible replies, $0.20 per 1,000 profiles, and $0.40 per 1,000 change records. The standard Apify apify-actor-start event is $0.00005 per run and covers the first five seconds of compute. There is no automatic dataset-item event. The delivery path checks the spending limit before each billable row, pushes it, then charges its post/profile/change event. Failed targets, retries, empty pages, filtered records and duplicates are not delivered as paid results.

A priced private run delivered 25 posts and one profile, with exactly 25 post-result and one profile-result events. Another run with a $0.001 charge limit discovered 26 unique records, requested ten after the result cap, delivered and charged two posts, then stopped; monitoring state stored only those two IDs. Event accounting and state were checked in RUN_SUMMARY and the named state store. Platform usage is included in this pricing configuration.

Verified cloud evidence and limits

• NASA profile: 100 unique posts plus profile across the public HTML window and six guest GraphQL pages. A separate maxPages: 2 run stopped after two cursor pages.
• Ten distinct profiles at concurrency six: 300 unique posts and ten profiles discovered across 28 pages. A global cap delivered 300 rows and reported ten excluded by cap. Zero failed or partial targets, one guest-token acquisition, 17.0 seconds wall time, about 133 MB peak RAM.
• A public post URL produced its root and three directly connected replies with stable relationship IDs.
• Staged 10/25/50/100 mixed-target tests reached 58/165/263/378 delivered rows. Those older workloads included unsupported targets and therefore had failures/partial targets; they do not establish 1,000+ unique records.

X can change its public web payloads and guest operations. Deep pagination is verified for profile timelines on several public accounts, not complete history or every account. Search, follower/following lists, X Lists, separate reply/media/repost timelines, quote listings and full conversation traversal are deliberately outside the supported product. Residential proxy cost and live 429 recovery are not proven; optional proxy use should be evaluated separately before Store publication. See docs/RELEASE_AUDIT.md for run IDs and evidence.

API example

curl -X POST "https://api.apify.com/v2/acts/yearly_register~x-twitter-scraper-dev/runs" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"targets":["@NASA"],"maxPosts":25}'

Keep the Apify API token in a secret or environment variable. The Actor itself requires no X login.