X / Twitter Scraper (Profiles & Tweets)

Scrape tweets from any public X/Twitter profile or individual tweet URL and get clean, structured JSON — text, engagement metrics, author details, media, hashtags, mentions, and links. This Actor drives a real Chromium browser and intercepts X's internal GraphQL API, so you get the same rich data the website itself receives, without relying on the paid X API.

Built on the Apify platform, it gives you scheduling, a REST API, integrations (Zapier, Make, Google Sheets, etc.), automatic proxy rotation, and run monitoring out of the box.

Why use this X / Twitter Scraper?

• Track brands, competitors, or public figures by pulling their latest tweets on a schedule.
• Social listening & research — collect tweets and their engagement for analysis or dashboards.
• Archive tweets before they disappear, with full metrics and media URLs.
• Feed downstream tools — export to JSON, CSV, or Excel, or pull via the Apify API.

How to use the X / Twitter Scraper

1. Add one or more profiles (handles like nasa, @apify, or full profile URLs) and/or tweet URLs.
2. Set Max tweets per profile and toggle Include replies if you want reply/thread coverage.
3. Leave Proxy configuration on the default (Apify Proxy — recommended and available on the free plan).
4. Click Start and download results from the Output/Storage tab when the run finishes.

One-time setup: authentication (required)

X requires a logged-in session to view almost all content. This Actor keeps authentication out of the input form — the operator supplies it once via secrets, so end users never touch credentials.

Recommended — auto-login (set it once, never refresh again). Store the account's login credentials as secrets and the Actor logs in for you, captures a fresh auth_token / ct0, saves them in a key-value store (twitter-session), reuses them on later runs, and re-logs in automatically when they expire:

apify secrets add x_email    "you@example.com"
apify secrets add x_username "your_handle"     # without the @
apify secrets add x_password 'your-password'

These map to X_EMAIL, X_USERNAME, X_PASSWORD in .actor/actor.json. The account must not have two-factor auth or e-mail/SMS login verification enabled — the Actor cannot solve those challenges and will abort with a clear log message if one appears (in that case, log in once manually, then fall back to the cookie method below).

Alternative — supply cookies manually. Get them from a browser logged into a throwaway X account (DevTools → Application → Cookies → x.com) and store them as a secret. Two options:

Option A — individual cookies:

apify secrets add twitter_auth_token "<auth_token cookie value>"
apify secrets add twitter_csrf_token "<ct0 cookie value>"

Option B — a full cookie bundle (JSON array of Playwright cookies, a {"name":"value"} map, or an a=1; b=2 header string):

$apify secrets add twitter_cookies '{"auth_token":"...","ct0":"..."}'

These are already wired to the TWITTER_AUTH_TOKEN, TWITTER_CSRF_TOKEN, and TWITTER_COOKIES environment variables in .actor/actor.json. Use a dedicated/disposable account — automated access can get accounts rate-limited or suspended.

Input

Configure the run from the Input tab or via JSON:

{
    "profiles": ["nasa", "@apify"],
    "tweetUrls": ["https://x.com/apify/status/1780000000000000000"],
    "maxTweetsPerProfile": 100,
    "includeReplies": false,
    "maxScrolls": 25,
    "proxyConfiguration": { "useApifyProxy": true }
}

Field	Type	Description
profiles	array	Handles (nasa, @apify) or profile URLs to scrape timelines from.
tweetUrls	array	Individual tweet/status URLs to scrape.
maxTweetsPerProfile	integer	Max tweets per profile (0 = unlimited, bounded by pagination).
includeReplies	boolean	Also load the "with replies" timeline / conversation replies.
maxScrolls	integer	Safety cap on scroll-based pagination per page.
proxyConfiguration	object	Proxy settings; defaults to Apify Proxy (recommended).

Output

Each item is one tweet. Download as JSON, CSV, Excel, or HTML from the dataset.

{
    "id": "1780000000000000000",
    "url": "https://x.com/apify/status/1780000000000000000",
    "text": "We just shipped a new feature!",
    "createdAt": "Wed Apr 10 12:34:56 +0000 2024",
    "lang": "en",
    "likeCount": 128,
    "retweetCount": 24,
    "replyCount": 9,
    "quoteCount": 3,
    "bookmarkCount": 12,
    "viewCount": 18452,
    "isRetweet": false,
    "isQuote": false,
    "isReply": false,
    "conversationId": "1780000000000000000",
    "hashtags": ["scraping"],
    "mentions": [],
    "urls": ["https://apify.com"],
    "media": [{ "type": "photo", "imageUrl": "https://pbs.twimg.com/media/....jpg", "videoUrl": null }],
    "authorUsername": "apify",
    "authorName": "Apify",
    "authorId": "123456",
    "authorVerified": true,
    "authorFollowers": 25000,
    "scrapedFrom": "profile:apify"
}

Data fields

Field	Description
id, url	Tweet ID and canonical URL.
text	Full tweet text (long-form note tweets included).
createdAt, lang, source	Timestamp, language, posting client.
likeCount, retweetCount, replyCount, quoteCount, bookmarkCount, viewCount	Engagement metrics.
isRetweet, isQuote, isReply	Tweet type flags.
conversationId, inReplyToUsername, inReplyToTweetId	Thread/reply context.
hashtags, mentions, urls, media	Extracted entities and media (image/video URLs).
authorUsername, authorName, authorId, authorVerified, authorFollowers, authorProfileImageUrl	Author details.

Pricing / cost estimation

This Actor is priced by Apify compute usage — you don't pay for a Twitter API subscription. Because it runs a browser, cost scales with how many tweets you scrape and how many scrolls are needed. To keep runs cheap: lower maxTweetsPerProfile and maxScrolls, and scrape only the profiles you need. Heavy resources (images, video, fonts) are blocked automatically to reduce proxy bandwidth.

Tips & advanced options

• Use a dedicated account for the cookie secret; rotate it if it gets challenged.
• Latest vs. exhausted timelines — the scraper stops after several idle scrolls (no new tweets), so unlimited runs still terminate.
• Proxies matter — keep Apify Proxy enabled; scraping X from a raw datacenter IP without proxying gets blocked quickly.
• Schedule it in the Apify Console for recurring monitoring.

FAQ, disclaimers & support

Is scraping X/Twitter legal? This Actor collects publicly available data. You are responsible for complying with X's Terms of Service, applicable laws (including GDPR/CCPA), and for not collecting private or personal data without a lawful basis. Use responsibly.

Why are my results empty? Almost always missing/expired authentication cookies, a protected account, or aggressive rate limiting. Refresh the cookie secret and keep proxying on.

Limitations — X changes its internal API frequently; if extraction drifts, open an issue. Very large timelines are bounded by how far X lets a session paginate.

For bugs or feature requests, use the Issues tab on the Actor page.