# X/Twitter Profile Scraper (`scrapesmith/x-twitter-profile-scraper`) Actor Scrape any public X (Twitter) profile. Extract every tweet with full text, likes, retweets, replies, views, media, hashtags and follower stats. Go beyond the tweet limit with deep history mode. Bulk scrape multiple profiles at once. - **URL**: https://apify.com/scrapesmith/x-twitter-profile-scraper.md - **Developed by:** [Scrape Smith](https://apify.com/scrapesmith) (community) - **Categories:** Social media, Automation, Developer tools - **Stats:** 20 total users, 17 monthly users, 100.0% runs succeeded, 1 bookmarks - **User rating**: No ratings yet ## Pricing from $0.15 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## X (Twitter) Profile Scraper Extract every public tweet from any X (Twitter) profile — full text, engagement metrics, media, and author details — and export the results as JSON, CSV, or Excel. Just enter a handle or profile URL and run. ### What It Does This actor scrapes the tweet history of any public X/Twitter profile. Give it a username (or a full profile URL) and it pulls tweets in reverse-chronological order, going far beyond the ~3,200-tweet limit most tools hit by using deep-history pagination to reach much further back into a profile's timeline. ### Key Features - **Deep tweet history** — collects well beyond Twitter's standard ~1,000–3,200 tweet profile cap - **Media tweets included** — photos, videos, and GIFs are picked up even when they don't appear on the main timeline - **Replies filtering** — choose to include or exclude reply tweets - **Bulk profile scraping** — scrape dozens or hundreds of profiles in a single run - **Resumable runs** — long scrapes checkpoint progress and can resume after a restart - **Structured, ready-to-use output** — clean JSON with engagement stats, hashtags, links, and mentions already parsed out ### Input Parameters | Field | Description | |---|---| | **Twitter/X Handles** | List of usernames to scrape (with or without `@`), e.g. `elonmusk`, `@nasa` | | **Profile URLs** | Direct profile URLs to scrape instead of/alongside handles, e.g. `https://x.com/elonmusk` | | **Max Tweets per Profile** | Maximum number of tweets to collect per profile (default: 100) | | **Include Replies** | Also collect tweets that are replies, not just original posts (default: off) | | **Max Total Items** | Optional hard cap on total items pushed across all profiles in the run | | **Search Fallback Mode** | Controls how the actor digs past the standard profile wall for older tweets — recommended default works well out of the box | | **Enable Sub-Tabs Fallback** | Also checks a profile's Media and Replies tabs to surface tweets missed on the main timeline (recommended, on by default) | | **Debug Logging** | Turn on for detailed diagnostic logs (off by default) | ### Output Fields Each scraped tweet includes: - `id` — tweet ID - `url` — direct link to the tweet - `text` — full tweet text - `createdAt` — publish date/time - `author` — user object (username, display name, bio, followers, verified status, profile picture, etc.) - `likeCount`, `retweetCount`, `replyCount`, `quoteCount`, `viewCount`, `bookmarkCount` — engagement metrics - `media` — attached photos/videos with direct URLs - `hashtags`, `mentions`, `urls` — extracted from the tweet text - `isRetweet`, `isReply`, `isQuote` — tweet type flags - `quotedTweet` — full details of a quoted tweet, if any - `lang` — detected language - `source` — app used to post (e.g. "Twitter for iPhone") ### Pricing This actor uses **pay-per-result** pricing — you're charged only for the tweets and profile records actually delivered to your dataset. No charge for failed or skipped profiles. ### Example Use Cases - **Social media monitoring** — track a competitor's or influencer's posting activity over time - **Market & brand research** — analyze what a company or public figure says about your industry - **Content archiving** — back up a profile's full tweet history - **Academic & data research** — build datasets of public discourse on a topic or account - **Lead generation** — identify engaged followers and frequent posters in a niche # Actor input Schema ## `twitterHandles` (type: `array`): List of Twitter/X usernames to scrape (with or without @). The scraper uses time-window pagination to go far beyond Twitter's standard ~1000-tweet profile cap. ## `startUrls` (type: `array`): Direct Twitter/X profile URLs to scrape (e.g. https://x.com/elonmusk). ## `maxTweets` (type: `integer`): Maximum number of tweets to collect per profile. Set to 5000+ to leverage the deep-history fallback. ## `includeReplies` (type: `boolean`): Also include tweets that are replies, not just original tweets. ## `maxItems` (type: `integer`): Hard cap on total items pushed to the dataset across all profiles combined. Leave empty to use the plan's default limit. ## `searchFallbackMode` (type: `string`): How the SearchTimeline fallback walks the user's history when the primary ~1000-tweet wall is hit. 'after-primary' (default) anchors windows at the oldest tweet the primary loop returned — eliminates the dupe-overlap that wastes paid requests. 'from-today' uses the original today-anchored behavior (higher cost, max coverage if you suspect the primary missed recent items). 'disabled' skips search fallback entirely (primary + sub-tabs only). ## `subTabsFallback` (type: `boolean`): Before the search fallback, scrape the user's separate Media and (when not already covered) Tweets-and-Replies sub-tabs. These have their own per-tab walls and often surface tweets the main timeline missed — no search filters required, no extra cost beyond the actual page requests. ## Actor input object example ```json { "twitterHandles": [ "elonmusk", "@nasa" ], "startUrls": [ { "url": "https://x.com/nasa" } ], "maxTweets": 5000, "includeReplies": false, "maxItems": 10000, "searchFallbackMode": "after-primary", "subTabsFallback": true } ```` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "twitterHandles": [ "elonmusk" ] }; // Run the Actor and wait for it to finish const run = await client.actor("scrapesmith/x-twitter-profile-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "twitterHandles": ["elonmusk"] } # Run the Actor and wait for it to finish run = client.actor("scrapesmith/x-twitter-profile-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "twitterHandles": [ "elonmusk" ] }' | apify call scrapesmith/x-twitter-profile-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=scrapesmith/x-twitter-profile-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "X/Twitter Profile Scraper", "description": "Scrape any public X (Twitter) profile. Extract every tweet with full text, likes, retweets, replies, views, media, hashtags and follower stats. Go beyond the tweet limit with deep history mode. Bulk scrape multiple profiles at once.", "version": "0.0", "x-build-id": "25xZz9dy48J8YAA2m" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/scrapesmith~x-twitter-profile-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-scrapesmith-x-twitter-profile-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/scrapesmith~x-twitter-profile-scraper/runs": { "post": { "operationId": "runs-sync-scrapesmith-x-twitter-profile-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/scrapesmith~x-twitter-profile-scraper/run-sync": { "post": { "operationId": "run-sync-scrapesmith-x-twitter-profile-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "twitterHandles": { "title": "Twitter / X Handles", "type": "array", "description": "List of Twitter/X usernames to scrape (with or without @). The scraper uses time-window pagination to go far beyond Twitter's standard ~1000-tweet profile cap.", "items": { "type": "string" } }, "startUrls": { "title": "Profile URLs", "type": "array", "description": "Direct Twitter/X profile URLs to scrape (e.g. https://x.com/elonmusk).", "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "maxTweets": { "title": "Max Tweets per Profile", "minimum": 1, "maximum": 500000, "type": "integer", "description": "Maximum number of tweets to collect per profile. Set to 5000+ to leverage the deep-history fallback.", "default": 10 }, "includeReplies": { "title": "Include Replies", "type": "boolean", "description": "Also include tweets that are replies, not just original tweets.", "default": false }, "maxItems": { "title": "Max Total Items", "minimum": 1, "maximum": 1000000, "type": "integer", "description": "Hard cap on total items pushed to the dataset across all profiles combined. Leave empty to use the plan's default limit.", "default": 10 }, "searchFallbackMode": { "title": "Search Fallback Mode", "enum": [ "after-primary", "from-today", "disabled" ], "type": "string", "description": "How the SearchTimeline fallback walks the user's history when the primary ~1000-tweet wall is hit. 'after-primary' (default) anchors windows at the oldest tweet the primary loop returned — eliminates the dupe-overlap that wastes paid requests. 'from-today' uses the original today-anchored behavior (higher cost, max coverage if you suspect the primary missed recent items). 'disabled' skips search fallback entirely (primary + sub-tabs only).", "default": "after-primary" }, "subTabsFallback": { "title": "Enable Sub-Tabs Fallback (Media + Tweets-and-Replies)", "type": "boolean", "description": "Before the search fallback, scrape the user's separate Media and (when not already covered) Tweets-and-Replies sub-tabs. These have their own per-tab walls and often surface tweets the main timeline missed — no search filters required, no extra cost beyond the actual page requests.", "default": true } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```