# X.com (Twitter) Community Post Scraper (`lurkapi/x-twitter-community-post-scraper`) Actor Scrape posts from any X.com (formerly Twitter) Community. Paste a community URL and get thousands of confirmed community posts straight to your spreadsheet, CRM, or dashboard. - **URL**: https://apify.com/lurkapi/x-twitter-community-post-scraper.md - **Developed by:** [LurkAPI](https://apify.com/lurkapi) (community) - **Categories:** Agents, Automation, Social media - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 1 bookmarks - **User rating**: No ratings yet ## Pricing from $0.30 / 1,000 post scrapeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## X.com (Twitter) Community Post Scraper Scrape posts from any X.com (formerly Twitter) Community. Paste a community URL and get thousands of confirmed community posts straight to your spreadsheet, CRM, or dashboard. ### 🐦 What it does Give it one or more community URLs (or 19-digit community IDs) and this tool returns every post it can confirm belongs to that community. For every post you get: - The full post text (no 280-character truncation on long-form notes) - Author handle, display name, and follower and following counts - Likes, retweets, quotes, replies, and view counts - Attached photos and videos (turn on the download add-on to save the files) - Precise creation timestamp in ISO 8601 - The community ID the post belongs to ### 📋 How to use it #### Step 1: Provide community URLs or IDs Paste one or more of these into the input: - Community URL: `https://x.com/i/communities/1493446837214187523` - Bare 19-digit ID: `1493446837214187523` Grab either from any X community page. Click "Communities" in the X sidebar, open a community, and copy the URL bar or the number after `/i/communities/`. You can mix URLs and IDs in one run. #### Step 2: Set the limit and sort - **Max posts per run**: hard cap on how many posts the run returns across every input community. Default is 100 to protect against surprise bills. Raise as high as you need. On an active community, expect around 1,000 to 1,500 posts per run. - **Sort posts by**: pick Newest for chronological or Most engaged for likes plus retweets first. #### Step 3: (Optional) Filter your results - **Minimum likes, retweets, replies, or views**: drop posts below your threshold. - **Exclude replies / retweets**: keep timelines clean. - **Must have media**: keep only posts with an attached photo or video. Filters are free. Posts that get filtered out are never billed. #### Step 4: (Optional) Add-ons Every add-on is off by default. Each one is billed only when it returns a real result. Failed lookups cost nothing. - **Community metadata**: adds one row per community with the community's rules, trending hashtags, member count, creator, banner, join policy, and topic. Billed once per community successfully resolved. - **Download media files**: saves every photo and video attached to a post into your run's key-value store. You get a signed download link on each media item. Billed once per post that had at least one file saved. #### Step 5: Run Click Start. Rows stream into the Output tab as each round finishes, so you do not have to wait for the whole run. ### 📊 What you get Every row has a `rowType` column so you can tell at a glance whether it is a `post` or a `community` metadata row. Four tabs group the fields by what you probably care about: | Tab | What is in it | |---|---| | 🔎 Overview | Row type, status, community name, URL, text, author, top counts, date | | 📝 Posts | Full post spread: every engagement count, media, author block, community ID | | 🏛️ Communities | Community metadata rows (with the metadata add-on): name, description, member count, rules, trending hashtags, creator, banner | | 🎬 Media | Posts that have media, with a signed download link per file when the download add-on is on | You can export from the Output tab as JSON, CSV, or Excel. Every row has a `status` field. `Success` means the row is good. Anything else is a human-readable message explaining what happened (community not found, community private, X temporarily unavailable). Rows that did not succeed are never billed. ### 💰 Pricing You only pay for what you get. Every event is billed per result. | Event | Rate | When it fires | |---|---|---| | Actor run start | $0.00005 | Once per run | | Post scraped | $0.30 / 1,000 | Once per unique post row | | Community metadata (add-on) | $0.02 per community | Once per community successfully resolved with the metadata add-on on | | Media download (add-on) | $2.00 / 1,000 | Once per post with at least one file saved | ### 💡 Good to know - **Long text is never truncated.** Long-form notes come back in full. - **URL forms accepted**: `x.com/i/communities/` and the bare 19-digit ID both work. - **On active communities, expect roughly 1,000 to 1,500 posts per run.** Very small or brand new communities may return fewer. - **Media files are stored in your Apify key-value store.** How long they are kept depends on your Apify plan. - **Rows appear in real time.** You can open the Output tab while the run is still going and start pulling data. ### For developers You can call this actor through the Apify API: ```json { "communityUrls": [ "https://x.com/i/communities/1493446837214187523", "1699807431709041070" ], "maxItems": 500, "sortPostsBy": "Newest", "minLikes": 10, "excludeReplies": true, "extractCommunityMetadata": true, "extractMedia": true } ```` Provide at least one entry in `communityUrls`. All other fields are optional. ### ❓ FAQ **How many posts do I get per community?** On an active community, around 1,000 to 1,500 posts per run. Very small or brand new communities return fewer, since there are simply fewer active posters. **Can I filter by author, hashtag, or date range?** You can filter by minimum likes, retweets, replies, and views. You can exclude replies or retweets and require attached media. You can sort by newest or most engaged. Author and date filters are not supported today. **Can I use my own X cookies?** No. This actor does not accept user cookies. **Why does one of my communities return fewer posts than another?** Community post yield depends on how many people post in it and how public their profiles are. A tightly moderated small community will return fewer posts than a large active one. **How long are downloaded media files stored?** Files land in your Apify key-value store. Retention depends on your Apify plan. ### 🔗 Other tools you might like - [X.com (Twitter) Tweet Scraper](https://apify.com/lurkapi/x-twitter-tweet-scraper): scrape tweets by handle, URL, or search query. - [X.com (Twitter) Thread Unroller & Scraper](https://apify.com/lurkapi/x-twitter-thread-unroller): expand full conversation threads. - [X (Twitter) Video, Photo & GIF Downloader](https://apify.com/lurkapi/x-twitter-video-photo-gif-downloader): download photos, videos, and GIFs from any tweet. - [X.com (Twitter) Profile Scraper](https://apify.com/lurkapi/x-twitter-profile-scraper): pull full profile data for any handle. - [X.com (Twitter) Tweet Replies Scraper](https://apify.com/lurkapi/x-twitter-tweet-replies-scraper): grab every reply to any tweet. ### ⚖️ Disclaimer This tool is intended for personal, research, and educational use. You are responsible for complying with X.com's Terms of Service and applicable laws in your jurisdiction. The developer is not liable for misuse. Data availability depends on X at run time. **Keywords:** twitter community scraper, x community scraper, twitter communities, x communities, community posts, community scraper, twitter scraper, x.com scraper # Actor input Schema ## `communityUrls` (type: `array`): X Community URLs (https://x.com/i/communities/1493446837214187523) or 19-digit numeric IDs. One entry per community. Mix URLs and IDs freely. ## `maxItems` (type: `integer`): Hard cap on total posts returned across all input communities. Default 100. ## `sortPostsBy` (type: `string`): How to order the post rows for each community. Newest = chronological (default). Most engaged = highest likes+retweets first. ## `minLikes` (type: `integer`): Drop posts below this favorite count. ## `minRetweets` (type: `integer`): Drop posts below this retweet count. ## `minReplies` (type: `integer`): Drop posts below this reply count. ## `minViews` (type: `integer`): Drop posts below this view count. ## `excludeReplies` (type: `boolean`): Drop reply posts. ## `excludeRetweets` (type: `boolean`): Drop retweets. ## `mustHaveMedia` (type: `boolean`): Keep only posts with attached photo or video. ## `extractCommunityMetadata` (type: `boolean`): Add one row per community with name, description, member count, rules, trending hashtags, creator, banner image, join policy, and topic. ## `extractMedia` (type: `boolean`): Save every photo and video attached to each post into your Apify storage. Charged once per post that has at least one file saved. ## `outputId` (type: `boolean`): Numeric ID (community or post). ## `outputUrl` (type: `boolean`): Canonical URL. ## `outputFullText` (type: `boolean`): Full post text. ## `outputCreatedAt` (type: `boolean`): ISO-8601 creation timestamp. ## `outputReplyCount` (type: `boolean`): Reply count. ## `outputRetweetCount` (type: `boolean`): Retweet count. ## `outputLikeCount` (type: `boolean`): Favorite/like count. ## `outputQuoteCount` (type: `boolean`): Quote-tweet count. ## `outputViewCount` (type: `boolean`): View count. ## `outputAuthor` (type: `boolean`): Author block (handle, name). ## `outputMedia` (type: `boolean`): Media array (photos, videos) attached to post. ## `outputCommunityDescription` (type: `boolean`): Community description text. ## `outputCommunityRules` (type: `boolean`): Array of community rules. ## `outputCommunityHashtags` (type: `boolean`): Array of hashtags trending inside the community. ## `outputCommunityMembers` (type: `boolean`): Facepile member avatar URLs shown on the community page. ## `bloomPagesPerAuthor` (type: `integer`): Internal tuning knob. Do not set manually. ## `bloomRounds` (type: `integer`): Internal tuning knob. Do not set manually. ## Actor input object example ```json { "communityUrls": [ "https://x.com/i/communities/1493446837214187523" ], "maxItems": 30, "sortPostsBy": "Newest", "excludeReplies": false, "excludeRetweets": false, "mustHaveMedia": false, "extractCommunityMetadata": false, "extractMedia": false, "outputId": true, "outputUrl": true, "outputFullText": true, "outputCreatedAt": true, "outputReplyCount": true, "outputRetweetCount": true, "outputLikeCount": true, "outputQuoteCount": true, "outputViewCount": true, "outputAuthor": true, "outputMedia": true, "outputCommunityDescription": true, "outputCommunityRules": true, "outputCommunityHashtags": true, "outputCommunityMembers": true, "bloomPagesPerAuthor": 2, "bloomRounds": 6 } ``` # Actor output Schema ## `overview` (type: `string`): All rows. ## `posts` (type: `string`): Post rows. ## `communities` (type: `string`): Community metadata rows (add-on). ## `media` (type: `string`): Posts with attached media (add-on). ## `keyValueStore` (type: `string`): Downloaded media files (add-on). # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "communityUrls": [ "https://x.com/i/communities/1493446837214187523" ], "maxItems": 30 }; // Run the Actor and wait for it to finish const run = await client.actor("lurkapi/x-twitter-community-post-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "communityUrls": ["https://x.com/i/communities/1493446837214187523"], "maxItems": 30, } # Run the Actor and wait for it to finish run = client.actor("lurkapi/x-twitter-community-post-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "communityUrls": [ "https://x.com/i/communities/1493446837214187523" ], "maxItems": 30 }' | apify call lurkapi/x-twitter-community-post-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=lurkapi/x-twitter-community-post-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "X.com (Twitter) Community Post Scraper", "description": "Scrape posts from any X.com (formerly Twitter) Community. Paste a community URL and get thousands of confirmed community posts straight to your spreadsheet, CRM, or dashboard.", "version": "0.0", "x-build-id": "8I80QHiTr7mI5EgSO" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/lurkapi~x-twitter-community-post-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-lurkapi-x-twitter-community-post-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/lurkapi~x-twitter-community-post-scraper/runs": { "post": { "operationId": "runs-sync-lurkapi-x-twitter-community-post-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/lurkapi~x-twitter-community-post-scraper/run-sync": { "post": { "operationId": "run-sync-lurkapi-x-twitter-community-post-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "communityUrls": { "title": "Community URLs or IDs", "type": "array", "description": "X Community URLs (https://x.com/i/communities/1493446837214187523) or 19-digit numeric IDs. One entry per community. Mix URLs and IDs freely.", "items": { "type": "string" } }, "maxItems": { "title": "Max posts per run", "minimum": 1, "type": "integer", "description": "Hard cap on total posts returned across all input communities. Default 100.", "default": 100 }, "sortPostsBy": { "title": "Sort posts", "enum": [ "Newest", "MostEngaged" ], "type": "string", "description": "How to order the post rows for each community. Newest = chronological (default). Most engaged = highest likes+retweets first.", "default": "Newest" }, "minLikes": { "title": "Minimum likes", "minimum": 0, "type": "integer", "description": "Drop posts below this favorite count." }, "minRetweets": { "title": "Minimum retweets", "minimum": 0, "type": "integer", "description": "Drop posts below this retweet count." }, "minReplies": { "title": "Minimum replies", "minimum": 0, "type": "integer", "description": "Drop posts below this reply count." }, "minViews": { "title": "Minimum views", "minimum": 0, "type": "integer", "description": "Drop posts below this view count." }, "excludeReplies": { "title": "Exclude replies", "type": "boolean", "description": "Drop reply posts.", "default": false }, "excludeRetweets": { "title": "Exclude retweets", "type": "boolean", "description": "Drop retweets.", "default": false }, "mustHaveMedia": { "title": "Must have media", "type": "boolean", "description": "Keep only posts with attached photo or video.", "default": false }, "extractCommunityMetadata": { "title": "Community metadata ($)", "type": "boolean", "description": "Add one row per community with name, description, member count, rules, trending hashtags, creator, banner image, join policy, and topic.", "default": false }, "extractMedia": { "title": "Download media files ($)", "type": "boolean", "description": "Save every photo and video attached to each post into your Apify storage. Charged once per post that has at least one file saved.", "default": false }, "outputId": { "title": "id", "type": "boolean", "description": "Numeric ID (community or post).", "default": true }, "outputUrl": { "title": "url", "type": "boolean", "description": "Canonical URL.", "default": true }, "outputFullText": { "title": "fullText", "type": "boolean", "description": "Full post text.", "default": true }, "outputCreatedAt": { "title": "createdAt", "type": "boolean", "description": "ISO-8601 creation timestamp.", "default": true }, "outputReplyCount": { "title": "replyCount", "type": "boolean", "description": "Reply count.", "default": true }, "outputRetweetCount": { "title": "retweetCount", "type": "boolean", "description": "Retweet count.", "default": true }, "outputLikeCount": { "title": "likeCount", "type": "boolean", "description": "Favorite/like count.", "default": true }, "outputQuoteCount": { "title": "quoteCount", "type": "boolean", "description": "Quote-tweet count.", "default": true }, "outputViewCount": { "title": "viewCount", "type": "boolean", "description": "View count.", "default": true }, "outputAuthor": { "title": "author", "type": "boolean", "description": "Author block (handle, name).", "default": true }, "outputMedia": { "title": "media", "type": "boolean", "description": "Media array (photos, videos) attached to post.", "default": true }, "outputCommunityDescription": { "title": "description (metadata rows)", "type": "boolean", "description": "Community description text.", "default": true }, "outputCommunityRules": { "title": "rules (metadata rows)", "type": "boolean", "description": "Array of community rules.", "default": true }, "outputCommunityHashtags": { "title": "trendingHashtags (metadata rows)", "type": "boolean", "description": "Array of hashtags trending inside the community.", "default": true }, "outputCommunityMembers": { "title": "topMembers (metadata rows)", "type": "boolean", "description": "Facepile member avatar URLs shown on the community page.", "default": true }, "bloomPagesPerAuthor": { "title": "internal bloom pages", "minimum": 1, "maximum": 5, "type": "integer", "description": "Internal tuning knob. Do not set manually.", "default": 2 }, "bloomRounds": { "title": "internal bloom rounds", "minimum": 1, "maximum": 6, "type": "integer", "description": "Internal tuning knob. Do not set manually.", "default": 6 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```