Ultimate X Scraper

Production-grade X (formerly Twitter) data extraction and optional AI content intelligence engine for Apify.

Ultimate X Scraper is a high-performance, production-ready Actor built for reliable, structured data acquisition from X. Extract post content, author metadata, public profile timelines, and X search results with deterministic schema, automatic target resolution, and optional structured AI intelligence.

---

Capabilities & Key Features

• Flexible Target Resolution: Provide profile handles (@handle), direct post URLs (https://x.com/user/status/123), or search queries in a single unified input list.
• Profile Timelines: Extract recent posts, reposts, replies, and author engagement metrics for public X profiles.
• Direct Post Extraction: Extract metadata for individual post links with author profile data and engagement counts when available.
• X Search Results: Search public X posts for keywords, hashtags, cashtags, or search queries with chronological (Latest) or relevance (Top) sorting.
• Optional AI Intelligence: Automated post classification, sentiment analysis, promotion detection, entity extraction, and structured summarization.
• Resilience & Non-Blocking AI: AI analysis runs post-collection. If AI service is unconfigured or experiences temporary outages, post extraction completes cleanly with ai.analyzed = false.
• Deduplication & Target Normalization: Automatic deduplication across target inputs prevents redundant data collection.
• Clean Normalized Output: Consistent JSON dataset schema ready for export to JSON/CSV or downstream integration workflows.

---

Supported Target Inputs

You can supply targets via the unified targets list or using specialized input arrays:

Input Field	Type	Description
targets	Array<string>	Unified list of mixed targets: usernames (@jack), profile links, post URLs, or search queries ("bitcoin news").
usernames	Array<string>	Explicit list of X profile usernames/handles to collect timelines for.
tweetUrls	Array<string>	Explicit list of direct X post URLs to fetch.
searchQueries	Array<string>	Explicit list of search terms or search queries.

---

Quick Start Examples

Example 1: Single Direct Post

{
  "targets": [
    "https://x.com/jack/status/20"
  ]
}

Example 2: Public User Profile Timeline

{
  "usernames": [
    "jack"
  ],
  "maxTweetsPerUser": 20,
  "includeReplies": true,
  "includeRetweets": false
}

Example 3: Public X Search

{
  "searchQueries": [
    "artificial intelligence"
  ],
  "maxSearchResults": 50,
  "searchProduct": "Latest"
}

Example 4: Mixed Multi-Target Collection

{
  "targets": [
    "@jack",
    "https://x.com/jack/status/20",
    "crypto market"
  ],
  "maxTweetsPerUser": 20,
  "maxSearchResults": 20,
  "concurrency": 4
}

Example 5: Generic AI Intelligence Analysis

{
  "targets": [
    "https://x.com/jack/status/20"
  ],
  "aiAnalysis": true
}

Example 6: Targeted Brand AI Analysis (Tesla)

{
  "targets": [
    "elonmusk"
  ],
  "aiAnalysis": true,
  "targetBrand": "Tesla"
}

Example 7: Targeted Campaign AI Analysis (Tesla Model Y & Robotaxi)

{
  "targets": [
    "elonmusk"
  ],
  "aiAnalysis": true,
  "targetBrand": "Tesla",
  "campaignKeywords": [
    "Model Y",
    "Robotaxi"
  ]
}

---

Dataset Output Format

ACTUAL CAPTURED OUTPUT RECORD (Verified Run on Post 20)

Each item pushed to the default dataset adheres to a deterministic, production-ready schema:

{
  "schemaVersion": "1",
  "sourceType": "tweet",
  "query": null,
  "requestedUsername": null,
  "id": "20",
  "url": "https://x.com/jack/status/20",
  "text": "just setting up my twttr",
  "createdAt": "2006-03-21T20:50:14.000Z",
  "language": "en",
  "isReply": false,
  "isRetweet": false,
  "author": {
    "username": "jack",
    "name": "jack"
  },
  "metrics": {
    "likes": 308542,
    "replies": 17980
  },
  "media": [],
  "scrapedAt": "2026-08-09T18:50:00.000Z",
  "ai": {
    "analyzed": true,
    "classification": "other",
    "confidence": 0.9,
    "promotionDetected": false,
    "brandMentioned": false,
    "entities": [],
    "sentiment": "unknown",
    "relevance": "unknown",
    "evidence": [],
    "summary": "The post is a simple statement about setting up Twitter."
  }
}

---

Metric Availability & Disclosure

• Likes and Replies: Likes and replies are included when available.
• Additional Engagement Metrics: Additional engagement metrics (such as retweets, quotes, bookmarks, or view counts) may be available depending on the result context and data exposed by X.
• Transparency: Metrics unavailable for a specific item are omitted from the output object rather than fabricated as zero.

---

AI Content Intelligence

When aiAnalysis is set to true, collected records are enriched with structured AI intelligence:

• Classification: Categorizes post content (e.g. promotional, informational, announcement, opinion, news, engagement, other).
• Sentiment: Analyzes content tone (positive, neutral, negative, mixed, unknown).
• Promotion Detection: Detects commercial calls-to-action or promotional content (promotionDetected: true/false).
• Brand & Entity Extraction: Identifies mentioned entities and flags brand mentions (brandMentioned: true/false).
• Structured Summarization: Generates a concise content summary (summary).
• Analytical Metrics: Returns confidence level (confidence, 0.0 to 1.0) and relevance assessment (relevance).

---

Pricing & Metering

Ultimate X Scraper uses Pay-Per-Event (PPE) usage pricing:

• Scraped Result Event: $0.35 per 1,000 successfully scraped records ($0.00035 / record).
• AI Analyzed Result Event: +$3.00 per 1,000 successfully AI-analyzed records ($0.00300 / analyzed record).

Additive Billing Structure

AI intelligence is an optional enrichment. Event charges apply to requested and successfully produced results:

• 1,000 Scraped Records (AI OFF): 1,000 × $0.00035 = $0.35
• 1,000 Scraped Records (1,000 AI Analyzed): 1,000 × $0.00035 + 1,000 × $0.00300 = $3.35
• 1,000 Scraped Records (200 AI Analyzed): 1,000 × $0.00035 + 200 × $0.00300 = $0.95

Usage-Based Event Metering

• Scraping Events: Scraping events are metered for successfully delivered results. Invalid or rejected inputs that produce no output generate zero scraped-result events.
• AI Analysis Events: AI analysis events are metered only for records successfully enriched by AI (ai.analyzed = true). If AI analysis is degraded or unconfigured (ai.analyzed = false), zero AI-analyzed events are charged.

---

Common Use Cases

1. Brand & Media Monitoring: Track brand mentions, public feedback, and content engagement across X.
2. Competitive Intelligence: Analyze public posting cadence, audience response, and content strategies of market competitors.
3. Market & Trend Research: Study conversation sentiment and topic volume surrounding emerging industry trends or public news.
4. Creator & Public Profile Analysis: Review public post metrics and timeline activity for featured accounts.
5. AI / ML Pipeline Integration: Extract structured social datasets formatted for language model ingestion, sentiment analytics, or automated reporting.

---

Error Handling & Run Statuses

At run completion, an executive run summary is written to Key-Value Store key OUTPUT:

{
  "schemaVersion": "1",
  "status": "SUCCEEDED",
  "targetsReceived": 3,
  "targetsSucceeded": 3,
  "targetsFailed": 0,
  "tweetsScraped": 42,
  "scrapedResultCount": 42,
  "aiAnalyzedResultCount": 42,
  "durationMs": 3450,
  "timestamp": "2026-08-09T18:50:00.000Z"
}

Standard Error Codes

If a target fails, it is recorded in summary.errors using standardized error codes:

• ERR_X101: Invalid input specification format.
• ERR_X201: Profile handle or account not found.
• ERR_X202: Post or Tweet URL not found.
• ERR_X301: Temporary rate limit encountered.
• ERR_X302: Target content restricted or protected.
• ERR_X305: Temporary collection runtime error.
• ERR_X501: Network connection timeout.

---

Limitations

• Public Data Only: Extracts publicly available posts, timelines, and search results. Protected accounts or private posts cannot be retrieved.
• Platform Variability: Data fields and engagement metric availability depend on public information provided by X for a given post or profile.
• Content Availability: Posts deleted or restricted by X will not be returned in datasets.

---

Downstream Integration & Export

Dataset results can be exported to JSON or CSV format, or integrated with downstream analytics pipelines and automation workflows via the Apify API, Python SDK, or Node.js client.

---

Responsible Use & Compliance

Users are responsible for ensuring that their data extraction activities comply with applicable privacy laws, data protection regulations, platform terms of service, and ethical scraping practices.

---

License

MIT License