# X.com Hashtag, Search & Profile Scraper (`dtrungtin/x-hashtag-search-profile-scraper`) Actor Extract X.com (Twitter) posts from any hashtag, search, or profile page with full metadata: text, authors, stats, media, and hashtags. - **URL**: https://apify.com/dtrungtin/x-hashtag-search-profile-scraper.md - **Developed by:** [Tin](https://apify.com/dtrungtin) (community) - **Categories:** - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $30.00 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## X.com Hashtag, Search & Profile Scraper Extract **X.com (Twitter) posts** from any hashtag page, search query, or user profile without dealing with X's anti-bot defenses, sign-in walls, or undocumented APIs. Paste one or more hashtag, search, or profile URLs and get back a **structured dataset of posts with full metadata**: text, authors, like/repost/reply/view counts, media, hashtags, and direct post URLs. Download your data as JSON, CSV, Excel, HTML, or XML directly from the Apify console, or fetch it through the Apify API. Schedule recurring runs, plug the output into integrations (Zapier, Make, webhooks), and monitor everything from one place. ### What you can do with the data - **Trend discovery** — track which posts are gaining traction inside any hashtag or search query, who their authors are, and how engagement (views, likes, reposts) scales over time. - **Creator & influencer research** — identify rising accounts inside a niche by sorting results by view count or engagement ratio. - **Brand & competitive monitoring** — run scheduled scrapes of hashtags and searches relevant to your brand or competitor campaigns and watch the conversation shift. - **Dataset building** — collect labeled post data (text, hashtags, media, stats) for ML training or content classification. - **Audience research** — opt in to `repliesPerPost` to pull reply text + replier handles alongside each post, useful for sentiment analysis, FAQ mining, or surfacing high-intent users in a niche. ### How to scrape X.com hashtags, searches, and profiles 1. Open the Actor and go to the **Input** tab. 2. Paste one or more X.com URLs into **Start URLs** — hashtag pages, search result pages, or profiles. 3. Set **Max items per URL** to the number of posts you want. 4. (Optional) Provide your own **auth token** for hashtag and search URLs (see [Authentication](#authentication) below). 5. Click **Start** and watch the dataset fill up in the **Output** tab. ### Input | Field | Type | Required | Description | | ---------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `startUrls` | array | yes | X.com hashtag URLs (`https://x.com/hashtag/`), search URLs (`https://x.com/search?q=`), or profile URLs (`https://x.com/`). `twitter.com` URLs are accepted and normalized. Single-post URLs (`/status/`) are rejected at startup. | | `maxItems` | integer | yes | Max posts to collect per URL (default: 10, range: 1–500). | | `repliesPerPost` | integer | no | How many replies to fetch per post, embedded into the post record under `replies`. Default `0` (skip replies). Range: 0–20. Enabling this navigates to each post's page individually, which adds run time linearly. | | `countryCode` | string | no | Exit-node country for the Apify residential proxy. One of `US`, `DE`, `VN`, `FR`, `GB`. Default: `US`. | | `authToken` | string (secret) | no | Optional. Value of the `auth_token` cookie from a logged-in x.com session. Unlocks hashtag/search URLs and full profile timelines. When omitted, the actor falls back to its own server-side session (if configured). | #### Example input ```json { "startUrls": [ { "url": "https://x.com/hashtag/ai" }, { "url": "https://x.com/search?q=cooking%20tips" }, { "url": "https://x.com/NASA" } ], "maxItems": 50, "repliesPerPost": 10, "countryCode": "US" } ``` #### Authentication X.com login-gates its **search and hashtag** pages, so scraping them requires a logged-in session. There are two ways to provide one: - **Bring your own (optional).** Log in to x.com in your browser, open DevTools → Application/Storage → Cookies → `https://x.com`, copy the value of the `auth_token` cookie, and paste it into the `authToken` input (the field is stored encrypted). **Use a throwaway account** — automated access violates X's Terms of Service, and accounts used for scraping can be restricted or banned. - **Server-side fallback.** When `authToken` is omitted, the actor uses its own session (the secret `X_AUTH_TOKEN` environment variable configured by the actor developer), if set. **Profile pages work even without a session**: X serves a limited server-rendered preview — roughly the 5 most recent posts plus the pinned one, with abbreviated engagement counts and no pagination. ### Output Each scraped post is pushed to the dataset as one record. Multiple URLs run in parallel; the `sourceType`, `sourceHashtag`, `sourceQuery`, `sourceUsername`, and `sourceUrl` fields let you tell records apart downstream. #### Example output record ```json { "sourceType": "profile", "sourceHashtag": null, "sourceQuery": null, "sourceUsername": "NASA", "sourceUrl": "https://x.com/NASA", "id": "1958159183456789012", "url": "https://x.com/NASA/status/1958159183456789012", "text": "The Sun just released the strongest flare of the year so far ☀️", "lang": "en", "createdAt": "2026-08-20T15:04:11.000Z", "author": { "id": "11348282", "username": "NASA", "name": "NASA", "avatar": "https://pbs.twimg.com/profile_images/....jpg", "verified": true, "followers": 79000000, "following": 180 }, "stats": { "replyCount": 512, "retweetCount": 4021, "likeCount": 23890, "quoteCount": 310, "bookmarkCount": 1804, "viewCount": 2413855 }, "media": [ { "type": "video", "url": "https://pbs.twimg.com/ext_tw_video_thumb/....jpg", "width": 1280, "height": 720, "videoUrl": "https://video.twimg.com/ext_tw_video/....mp4", "durationMs": 23000 } ], "hashtags": ["solarflare"], "mentions": [], "links": ["https://www.nasa.gov/..."], "isRetweet": false, "isQuote": false, "isReply": false, "conversationId": "1958159183456789012", "inReplyToStatusId": null, "quotedStatusId": null, "replies": [ { "id": "1958160000000000000", "text": "Incredible footage!", "createdAt": "2026-08-20T15:09:42.000Z", "likeCount": 42, "replyCount": 1, "user": { "id": "44196397", "username": "spacefan", "name": "Space Fan", "avatar": "https://pbs.twimg.com/profile_images/....jpg", "verified": false, "followers": 1200, "following": 300 } } ] } ``` The `replies` array is only present when `repliesPerPost > 0`. When it's `0` (the default), the field is omitted. #### Field reference - **`sourceType` / `sourceHashtag` / `sourceQuery` / `sourceUsername` / `sourceUrl`** — which input URL produced this record (set per item so concurrent scrapes don't get jumbled). `sourceType` is `"hashtag"`, `"search"`, or `"profile"`; `sourceHashtag` is populated only for hashtag URLs, `sourceQuery` only for search URLs, `sourceUsername` only for profile URLs. - **`id` / `url`** — post ID and canonical web URL. - **`text` / `createdAt` / `lang`** — full post text (long posts are expanded from the notes format), ISO timestamp, and detected language. - **`author`** — poster's ID, `@handle` (`username`), display name, avatar, verified flag, and follower/following counts at scrape time. - **`stats`** — reply / repost (`retweetCount`) / like / quote / bookmark / view counts at scrape time. - **`media`** — photos, videos, and GIFs attached to the post. Videos and GIFs include the highest-bitrate MP4 `videoUrl` and `durationMs`. - **`hashtags` / `mentions` / `links`** — parsed post entities (hashtag names, mentioned handles, expanded outbound URLs). - **`isRetweet` / `isQuote` / `isReply`** — post-type flags, with `conversationId`, `inReplyToStatusId`, `quotedStatusId` for joining threads downstream. - **`replies`** — (optional, only when `repliesPerPost > 0`) top-level replies fetched by visiting each post's page. Reply-to-reply threads are not included. ### How it works This actor uses `PlaywrightCrawler` (Crawlee) with [Camoufox](https://github.com/daijro/camoufox) — a hardened, stealth-focused Firefox build — to mimic a real desktop browser. Per request it: 1. Navigates to the target page and intercepts the relevant internal GraphQL response — `SearchTimeline` for hashtag and search pages, `UserTweets` for profile pages — to extract the structured post list. Ads (promoted posts) are filtered out. 2. Scrolls to load more posts until either `maxItems` is reached or the timeline reports no more results. 3. (Optional) If `repliesPerPost > 0`, navigates the same browser session to each collected post's page, intercepts `TweetDetail`, scrolls the reply thread, and embeds the captured replies into that post's record before pushing to the dataset. Unrelated "Discover more" suggestions are filtered out by conversation ID. Heavy resources (images, media, fonts) and tracking hosts are blocked at the request level to cut proxy bandwidth and speed up the run. ### How much does it cost to scrape X.com? The actor uses a residential proxy and a real browser, so cost scales with the number of posts and especially with `repliesPerPost` (each post needs its own page navigation). A typical run collecting a few hundred posts finishes in a few minutes. Start with small `maxItems` values, check the results, then scale up. ### Tips - **Profiles work even without a session** — but only the latest ~5 posts. For deeper profile timelines, the actor's logged-in session is used automatically when configured. - **Use `&f=live` in search URLs** (e.g. `https://x.com/search?q=apify&f=live`) to get the *Latest* tab in chronological order instead of *Top*. - **Keep `repliesPerPost` small.** Each post costs one extra navigation plus scrolling; 50 posts × 20 replies can add many minutes. - **Region-locked trends.** Set `countryCode` to pin reproducible results when content differs by region. ### Notes and limitations - **Hashtag, search, and profile URLs only.** Single-post URLs (`/status/`) are rejected at startup. - **Search and hashtags require a logged-in session.** Provide the `authToken` input, or the actor falls back to its server-side session (`X_AUTH_TOKEN`). If neither is available (or the session expired), X.com redirects those pages to the login screen and the actor logs a clear error and skips the URL. - **Logged-out profile limits.** Without a session, X serves a server-rendered preview with only the ~5 most recent posts (plus the pinned one). The actor parses those directly: post text, exact timestamps (derived from the post ID), media, and links are accurate, but engagement counts are approximate (X abbreviates them, e.g. `6.8K`), `quoteCount`/`bookmarkCount`/follower counts are unavailable, and the replies phase is skipped. With the session configured, full timelines with exact stats are scraped. - **Signed media URLs can expire.** Avatar and video URLs served by X's CDN may stop working after some time — download promptly if you need the bytes. - **View counts are approximate.** X rounds large view counts on the web client. - **Respect the law and X's rules.** Only scrape publicly available data, comply with GDPR/CCPA and X.com's Terms of Service, and do not collect or store personal data without a lawful basis. You are responsible for how you use the output. ### Related Actors you might like Speed up your whole scraping workflow with these companion tools from the same developer: - **[TikTok Hashtag, Search & Profile Scraper](https://apify.com/dtrungtin/tiktok-hashtag-search-profile-scraper)** — The same three-mode scraper for TikTok: hashtags, search queries, and profiles in, structured video data out. - **[AI Dataset Search](https://apify.com/dtrungtin/ai-dataset-search)** — Search and explore your scraped datasets in plain English. Ask questions and instantly surface the records you need from the output of this and other Actors. - **[AI Scraper Input Generator](https://apify.com/dtrungtin/ai-scraper-input-generator)** — Turn a plain-language description of what you want to scrape into a ready-to-run input configuration. ### Contact Found a bug or want a feature? Open an issue in the Actor's **Issues** tab or email . For more scrapers, browse the [Apify Store](https://apify.com/store). # Actor input Schema ## `startUrls` (type: `array`): X.com hashtag URLs (https://x.com/hashtag/), search URLs (https://x.com/search?q=), or profile URLs (https://x.com/). twitter.com URLs are accepted too. Other URL types (single posts, lists) are not supported. ## `maxItems` (type: `integer`): Maximum number of posts to collect per hashtag, search, or profile URL. ## `repliesPerPost` (type: `integer`): How many replies to fetch per post (embedded into each post record under `replies`). Set to 0 to skip replies entirely. Note: enabling this navigates to each post's page individually, so a large value across many posts significantly increases run time. ## `countryCode` (type: `string`): Country for the residential proxy exit node. Use this when the target site serves different content by region. ## `authToken` (type: `string`): Value of the `auth_token` cookie from a logged-in x.com browser session. Unlocks hashtag and search URLs and full profile timelines. When omitted, the actor falls back to its own server-side session (if configured). Use a throwaway account - automated access may get the account restricted. ## Actor input object example ```json { "startUrls": [ { "url": "https://x.com/NASA" } ], "maxItems": 10, "repliesPerPost": 0, "countryCode": "US" } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://x.com/NASA" } ], "maxItems": 10, "repliesPerPost": 0 }; // Run the Actor and wait for it to finish const run = await client.actor("dtrungtin/x-hashtag-search-profile-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://x.com/NASA" }], "maxItems": 10, "repliesPerPost": 0, } # Run the Actor and wait for it to finish run = client.actor("dtrungtin/x-hashtag-search-profile-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://x.com/NASA" } ], "maxItems": 10, "repliesPerPost": 0 }' | apify call dtrungtin/x-hashtag-search-profile-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,dtrungtin/x-hashtag-search-profile-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/eK39V5fsSCmMIn3YU/builds/CqQdmw5yyEhmUKJua/openapi.json