X.com Hashtag, Search & Profile Scraper

Extract X.com (Twitter) posts from any hashtag page, search query, or user profile without dealing with X's anti-bot defenses, sign-in walls, or undocumented APIs. Paste one or more hashtag, search, or profile URLs and get back a structured dataset of posts with full metadata: text, authors, like/repost/reply/view counts, media, hashtags, and direct post URLs.

Download your data as JSON, CSV, Excel, HTML, or XML directly from the Apify console, or fetch it through the Apify API. Schedule recurring runs, plug the output into integrations (Zapier, Make, webhooks), and monitor everything from one place.

What you can do with the data

• Trend discovery — track which posts are gaining traction inside any hashtag or search query, who their authors are, and how engagement (views, likes, reposts) scales over time.
• Creator & influencer research — identify rising accounts inside a niche by sorting results by view count or engagement ratio.
• Brand & competitive monitoring — run scheduled scrapes of hashtags and searches relevant to your brand or competitor campaigns and watch the conversation shift.
• Dataset building — collect labeled post data (text, hashtags, media, stats) for ML training or content classification.
• Audience research — opt in to repliesPerPost to pull reply text + replier handles alongside each post, useful for sentiment analysis, FAQ mining, or surfacing high-intent users in a niche.

How to scrape X.com hashtags, searches, and profiles

1. Open the Actor and go to the Input tab.
2. Paste one or more X.com URLs into Start URLs — hashtag pages, search result pages, or profiles.
3. Set Max items per URL to the number of posts you want.
4. (Optional) Provide your own auth token for hashtag and search URLs (see Authentication below).
5. Click Start and watch the dataset fill up in the Output tab.

Input

Field	Type	Required	Description
startUrls	array	yes	X.com hashtag URLs (https://x.com/hashtag/<name>), search URLs (https://x.com/search?q=<query>), or profile URLs (https://x.com/<username>). twitter.com URLs are accepted and normalized. Single-post URLs (/status/<id>) are rejected at startup.
maxItems	integer	yes	Max posts to collect per URL (default: 10, range: 1–500).
repliesPerPost	integer	no	How many replies to fetch per post, embedded into the post record under replies. Default 0 (skip replies). Range: 0–20. Enabling this navigates to each post's page individually, which adds run time linearly.
countryCode	string	no	Exit-node country for the Apify residential proxy. One of US, DE, VN, FR, GB. Default: US.
authToken	string (secret)	no	Optional. Value of the auth_token cookie from a logged-in x.com session. Unlocks hashtag/search URLs and full profile timelines. When omitted, the actor falls back to its own server-side session (if configured).

Example input

{
    "startUrls": [
        { "url": "https://x.com/hashtag/ai" },
        { "url": "https://x.com/search?q=cooking%20tips" },
        { "url": "https://x.com/NASA" }
    ],
    "maxItems": 50,
    "repliesPerPost": 10,
    "countryCode": "US"
}

Authentication

X.com login-gates its search and hashtag pages, so scraping them requires a logged-in session. There are two ways to provide one:

• Bring your own (optional). Log in to x.com in your browser, open DevTools → Application/Storage → Cookies → https://x.com, copy the value of the auth_token cookie, and paste it into the authToken input (the field is stored encrypted). Use a throwaway account — automated access violates X's Terms of Service, and accounts used for scraping can be restricted or banned.
• Server-side fallback. When authToken is omitted, the actor uses its own session (the secret X_AUTH_TOKEN environment variable configured by the actor developer), if set.

Profile pages work even without a session: X serves a limited server-rendered preview — roughly the 5 most recent posts plus the pinned one, with abbreviated engagement counts and no pagination.

Output

Each scraped post is pushed to the dataset as one record. Multiple URLs run in parallel; the sourceType, sourceHashtag, sourceQuery, sourceUsername, and sourceUrl fields let you tell records apart downstream.

Example output record

{
    "sourceType": "profile",
    "sourceHashtag": null,
    "sourceQuery": null,
    "sourceUsername": "NASA",
    "sourceUrl": "https://x.com/NASA",
    "id": "1958159183456789012",
    "url": "https://x.com/NASA/status/1958159183456789012",
    "text": "The Sun just released the strongest flare of the year so far ☀️",
    "lang": "en",
    "createdAt": "2026-08-20T15:04:11.000Z",
    "author": {
        "id": "11348282",
        "username": "NASA",
        "name": "NASA",
        "avatar": "https://pbs.twimg.com/profile_images/....jpg",
        "verified": true,
        "followers": 79000000,
        "following": 180
    },
    "stats": {
        "replyCount": 512,
        "retweetCount": 4021,
        "likeCount": 23890,
        "quoteCount": 310,
        "bookmarkCount": 1804,
        "viewCount": 2413855
    },
    "media": [
        {
            "type": "video",
            "url": "https://pbs.twimg.com/ext_tw_video_thumb/....jpg",
            "width": 1280,
            "height": 720,
            "videoUrl": "https://video.twimg.com/ext_tw_video/....mp4",
            "durationMs": 23000
        }
    ],
    "hashtags": ["solarflare"],
    "mentions": [],
    "links": ["https://www.nasa.gov/..."],
    "isRetweet": false,
    "isQuote": false,
    "isReply": false,
    "conversationId": "1958159183456789012",
    "inReplyToStatusId": null,
    "quotedStatusId": null,
    "replies": [
        {
            "id": "1958160000000000000",
            "text": "Incredible footage!",
            "createdAt": "2026-08-20T15:09:42.000Z",
            "likeCount": 42,
            "replyCount": 1,
            "user": {
                "id": "44196397",
                "username": "spacefan",
                "name": "Space Fan",
                "avatar": "https://pbs.twimg.com/profile_images/....jpg",
                "verified": false,
                "followers": 1200,
                "following": 300
            }
        }
    ]
}

The replies array is only present when repliesPerPost > 0. When it's 0 (the default), the field is omitted.

Field reference

• sourceType / sourceHashtag / sourceQuery / sourceUsername / sourceUrl — which input URL produced this record (set per item so concurrent scrapes don't get jumbled). sourceType is "hashtag", "search", or "profile"; sourceHashtag is populated only for hashtag URLs, sourceQuery only for search URLs, sourceUsername only for profile URLs.
• id / url — post ID and canonical web URL.
• text / createdAt / lang — full post text (long posts are expanded from the notes format), ISO timestamp, and detected language.
• author — poster's ID, @handle (username), display name, avatar, verified flag, and follower/following counts at scrape time.
• stats — reply / repost (retweetCount) / like / quote / bookmark / view counts at scrape time.
• media — photos, videos, and GIFs attached to the post. Videos and GIFs include the highest-bitrate MP4 videoUrl and durationMs.
• hashtags / mentions / links — parsed post entities (hashtag names, mentioned handles, expanded outbound URLs).
• isRetweet / isQuote / isReply — post-type flags, with conversationId, inReplyToStatusId, quotedStatusId for joining threads downstream.
• replies — (optional, only when repliesPerPost > 0) top-level replies fetched by visiting each post's page. Reply-to-reply threads are not included.

How it works

This actor uses PlaywrightCrawler (Crawlee) with Camoufox — a hardened, stealth-focused Firefox build — to mimic a real desktop browser. Per request it:

1. Navigates to the target page and intercepts the relevant internal GraphQL response — SearchTimeline for hashtag and search pages, UserTweets for profile pages — to extract the structured post list. Ads (promoted posts) are filtered out.
2. Scrolls to load more posts until either maxItems is reached or the timeline reports no more results.
3. (Optional) If repliesPerPost > 0, navigates the same browser session to each collected post's page, intercepts TweetDetail, scrolls the reply thread, and embeds the captured replies into that post's record before pushing to the dataset. Unrelated "Discover more" suggestions are filtered out by conversation ID.

Heavy resources (images, media, fonts) and tracking hosts are blocked at the request level to cut proxy bandwidth and speed up the run.

How much does it cost to scrape X.com?

The actor uses a residential proxy and a real browser, so cost scales with the number of posts and especially with repliesPerPost (each post needs its own page navigation). A typical run collecting a few hundred posts finishes in a few minutes. Start with small maxItems values, check the results, then scale up.

Tips

• Profiles work even without a session — but only the latest ~5 posts. For deeper profile timelines, the actor's logged-in session is used automatically when configured.
• Use &f=live in search URLs (e.g. https://x.com/search?q=apify&f=live) to get the Latest tab in chronological order instead of Top.
• Keep repliesPerPost small. Each post costs one extra navigation plus scrolling; 50 posts × 20 replies can add many minutes.
• Region-locked trends. Set countryCode to pin reproducible results when content differs by region.

Notes and limitations

• Hashtag, search, and profile URLs only. Single-post URLs (/status/<id>) are rejected at startup.
• Search and hashtags require a logged-in session. Provide the authToken input, or the actor falls back to its server-side session (X_AUTH_TOKEN). If neither is available (or the session expired), X.com redirects those pages to the login screen and the actor logs a clear error and skips the URL.
• Logged-out profile limits. Without a session, X serves a server-rendered preview with only the ~5 most recent posts (plus the pinned one). The actor parses those directly: post text, exact timestamps (derived from the post ID), media, and links are accurate, but engagement counts are approximate (X abbreviates them, e.g. 6.8K), quoteCount/bookmarkCount/follower counts are unavailable, and the replies phase is skipped. With the session configured, full timelines with exact stats are scraped.
• Signed media URLs can expire. Avatar and video URLs served by X's CDN may stop working after some time — download promptly if you need the bytes.
• View counts are approximate. X rounds large view counts on the web client.
• Respect the law and X's rules. Only scrape publicly available data, comply with GDPR/CCPA and X.com's Terms of Service, and do not collect or store personal data without a lawful basis. You are responsible for how you use the output.

Related Actors you might like

Speed up your whole scraping workflow with these companion tools from the same developer:

• TikTok Hashtag, Search & Profile Scraper — The same three-mode scraper for TikTok: hashtags, search queries, and profiles in, structured video data out.
• AI Dataset Search — Search and explore your scraped datasets in plain English. Ask questions and instantly surface the records you need from the output of this and other Actors.
• AI Scraper Input Generator — Turn a plain-language description of what you want to scrape into a ready-to-run input configuration.

Contact

Found a bug or want a feature? Open an issue in the Actor's Issues tab or email dtrungtin@gmail.com.

For more scrapers, browse the Apify Store.