X.com (Twitter.com) Scraper

A robust, async-ready Python Apify Actor for retrieving detailed Tweet metadata from X (Twitter) using only public Tweet IDs or URLs—no API key or user authentication required.

What does the X.com Scraper do?

The X.com Scraper extracts full Tweet metadata for any public Tweet, given either its tweet ID or a standard X/Twitter URL. This Actor is purpose-built for machine learning, data analytics, sentiment analysis, trend monitoring, digital marketing, and social media research, allowing you to automate collection of Tweet data at scale without relying on limited or expensive official APIs.

Why use the X.com Scraper?

• No API key required: Works with public Tweet IDs or URLs, bypassing API rate limits and credential requirements.
• Scalable Automation: Designed for batch scraping—input hundreds or thousands of Tweets per run.
• Structured Output: Returns rich, normalized JSON for each Tweet, ideal for pipelines, analytics, and ML ingestion.
• Handles Multiple Input Types: Accepts both direct Tweet IDs and full Tweet URLs in the same run.
• Async Performance: Fully async, enabling high concurrency for faster data gathering.
• Transparent Errors: Each result includes errors (if encountered) for robust downstream handling.
• Easy Integration: Standard Apify storage and dataset outputs.

Features

• Accepts both Tweet IDs and URLs: Mix and match in your input.
• Automatic ID extraction: Recognizes and parses IDs from any regular X/Twitter status URL.
• Async HTTP Requests: Efficiently fetches many Tweets in parallel.
• Rich Metadata: Retrieves all structured Tweet fields, including:
  • Text, user info, hashtags, mentions, media, timestamps, engagement counts, and more.
• Error Handling: Detailed info even for missing, deleted, or protected tweets.
• Apify SDK Integration: Leverages the Apify Python SDK for local or cloud runs, storage, and logging.
• Machine Learning Ready: Output is optimized for ML, data science, social analytics, and market research.
• Ideal for Web Analytics: Use for tracking brands, campaigns, topics, or market sentiment in real time.
• Open for Automation: Integrate easily with your own workflows, CRON jobs, or cloud automations.

Use Cases

• Machine Learning: Ingest Tweet data for language modeling, sentiment analysis, or fine-tuning.
• Web & Social Analytics: Track hashtag campaigns, trending topics, or competitor mentions.
• Sales & Brand Monitoring: Monitor customer feedback, complaints, and product buzz at scale.
• Digital Marketing: Aggregate results for engagement analysis, campaign reporting, or influencer research.
• Academic & Journalistic Research: Collect primary source social posts for large-scale, ethical studies.

How It Works

• Add your logic inside the async with Actor: block (see src/main.py).
• Input: Submit a list of Tweet IDs and/or direct Tweet URLs.
• The Actor:
  • Extracts valid Tweet IDs from each input.
  • Makes async HTTP requests to Twitter/X's public syndication endpoint.
  • Collects and normalizes JSON responses.
  • Stores everything in the Apify dataset for easy download or further processing.

Example

{"tweet_ids": ["719484841172054016", "https://twitter.com/user/status/1517493440075018240"]}