Twitter (X) Posts Search Scraper

Scrape Twitter/X posts by keyword without the official API. This actor runs an X advanced search for one or many keywords, auto-paginates the results, and exports every matching tweet as structured JSON — text, author, likes, retweets, replies, quotes, views, media, hashtags and timestamps. Results can be downloaded as JSON, CSV, Excel or accessed via the Apify API.

The output is built for pipelines: flat items with id_str, url, text, created_at, engagement counts and a full user author object, so it slots straight into existing data workflows.

Features

• 🔍 Keyword search — single keyword, exact phrases ("like this"), or many keywords per run, each searched separately and de-duplicated.
• 🗂 Latest & Top tabs — chronological or engagement-ranked results.
• 📅 Date filters — since/until bounds, applied as search operators and re-checked on every scraped tweet.
• 🎛 Content filters — links, hashtags, mentions, news, safe content, questions, verified / blue-verified authors, language.
• ♾ Auto-pagination — scrolls until your target tweet count is reached or the feed runs dry (up to 5,000 per run).
• 💾 Incremental results — tweets are pushed to the dataset as each keyword finishes, so long runs never lose collected data to a timeout.
• 🔐 Flexible authentication — session cookies, secret env vars, or fully automated login with session caching.
• 🛠 Debuggable failures — on empty runs the actor saves a screenshot and the page HTML to the run's key-value store.

Use cases

• Brand, product and competitor monitoring
• Sentiment analysis and social listening pipelines
• Market and academic research
• Lead generation and trend discovery

Quick start

1. Enter one or more keywords.
2. Provide a session (see Authentication below) — a throwaway X account's cookies or credentials.
3. Run. Tweets appear in the dataset as each keyword completes.

Input

Field	Type	Description
query	string	One keyword/phrase. Several keywords can be pasted one per line. Advanced-search operators (from:, "exact phrase", ...) pass through as-is.
queries	array	Multiple keywords — each searched separately, results combined and de-duplicated by tweet. numberOfTweets applies per keyword.
search_type	string	Latest (chronological) or Top (ranked). Default Latest.
numberOfTweets	integer	Target tweets per keyword (max 5000). Default 100.
scrapeAll	boolean	Auto-paginate until the target is reached. When false only the first page is returned. Default true.
timeSince / timeUntil	string	yyyy-mm-dd date bounds — applied as since:/until: operators and re-checked on the scraped tweets.
language	string	ISO 639-1 code, applied as lang:.
contentQuestionMarks	boolean	Questions only (? operator).
engagementHasEngagement	boolean	filter:has_engagement.
mediaHasHashtags / mediaHasLinks / mediaHasMentions	boolean	filter:hashtags / filter:links / filter:mentions.
mediaNewsOnly / mediaSafeContentOnly	boolean	filter:news / filter:safe.
usersBlueVerifiedOnly / usersVerifiedOnly	boolean	filter:blue_verified / filter:verified.
cookies	array	Cookies of a logged-in X session (Cookie-Editor/EditThisCookie JSON export). The auth_token + ct0 cookies carry the session.
twUsername / twPassword / twEmail	string	Credentials for automated login (2FA must be disabled). The email answers X's occasional identity check.
proxyConfiguration	object	Proxy settings. Residential proxies strongly recommended.
headless	boolean	Uncheck only for local debugging.

Authentication

X login-walls search for anonymous visitors, so the actor needs a session. It resolves one in this priority order:

1. the cookies input (per-run),
2. the TW_COOKIES secret env var (JSON cookie array or a name=value; name2=value2 header string),
3. the session cached in the twitter-session key-value store by a previous automated login,
4. automated login with twUsername/twPassword (or the TW_USERNAME/TW_PASSWORD/TW_EMAIL secret env vars) — the fresh session is cached for future runs.

Always use a throwaway account, never a personal one. Accounts used for scraping can be restricted by X.

Output

One dataset item per tweet:

{
    "id": "1234567890123456789",
    "id_str": "1234567890123456789",
    "url": "https://x.com/user/status/1234567890123456789",
    "twitterUrl": "https://twitter.com/user/status/1234567890123456789",
    "text": "Tweet text...",
    "full_text": "Tweet text...",
    "lang": "en",
    "created_at": "2026-05-01T09:30:00.000Z",
    "createdTime": "2026-05-01T09:30:00.000Z",
    "favorites": 12,
    "retweets": 3,
    "replies": 4,
    "quotes": 1,
    "bookmarks": 2,
    "views": 4520,
    "hashtags": ["health"],
    "mentions": ["someuser"],
    "urls": ["https://example.com/article"],
    "media": [{ "type": "photo", "url": "https://pbs.twimg.com/media/...", "expanded_url": "...", "video_url": null }],
    "is_retweet": false,
    "is_quote": false,
    "is_reply": false,
    "conversation_id_str": "1234567890123456789",
    "possibly_sensitive": false,
    "user": {
        "id_str": "987654321",
        "name": "User Name",
        "screen_name": "user",
        "url": "https://x.com/user",
        "followers": 1500,
        "following": 300,
        "tweets": 8200,
        "verified": false,
        "blue_verified": true,
        "avatar": "https://pbs.twimg.com/profile_images/...",
        "created_at": "2015-03-01T00:00:00.000Z",
        "location": "Harare"
    },
    "username": "user",
    "authorName": "User Name",
    "author": { "id": "987654321", "name": "User Name", "username": "user", "profileUrl": "https://x.com/user" },
    "searchQuery": "econet",
    "search_type": "Latest",
    "scrapedAt": "2026-07-03T12:00:00.000Z"
}

How it works

The actor opens x.com/search in a Playwright-driven Chrome with an authenticated session, intercepts the SearchTimeline GraphQL responses that X streams to the page, and scrolls to auto-paginate. Parsing walks the GraphQL JSON generically instead of hardcoding one payload path, so it survives X's frequent payload-shape changes.

FAQ & troubleshooting

Why do I get 0 results? Almost always an expired/missing session or a blocked proxy IP. Check DEBUG_SCREENSHOT and DEBUG_HTML in the run's key-value store to see exactly what the browser saw. Also verify the keyword actually has matches on x.com/search.

Login fails — what now? The error message says which check X raised (identity confirmation, confirmation code/2FA, Arkose CAPTCHA, suspension) and what to do about it. Logging the account in manually once from the proxy's region usually clears security checks. When in doubt, export cookies from a logged-in browser session and pass those instead.

Navigation timeouts? Use Apify Proxy with the RESIDENTIAL group; datacenter IPs are widely blocked by X.

Is this legal? The actor extracts only publicly visible data. You are responsible for complying with X's Terms of Service and applicable laws (e.g. GDPR) for your use case — consult a lawyer if unsure.