Twitter Advanced Search Scraper

A powerful and flexible Apify Actor for scraping Twitter/X data. Advanced X (Twitter) post search with 50+ filters: content, users, geo, time, engagement, media. Dual modes: simple queries or structured filters. Enterprise-grade reliability for precise data extraction.

🚀 Features

• Three Search Modes:
  • Global Search: Search for tweets across all of Twitter using keywords, hashtags, and filters.
  • Profile Search: Scrape tweets from a specific user's profile with date and keyword filtering.
  • User Search: Find user profiles based on search queries.
• Advanced Filtering: Filter by:
  • Engagement: Minimum likes, retweets, replies.
  • Media Type: Images, videos, links, news.
  • Content: Keywords, exact phrases, hashtags, cashtags.
  • Time: Date ranges (since/until) and relative time.
  • Location: Geolocation and radius search.
• Rich Data Extraction:
  • Full tweet text and metadata (ID, date, URL).
  • User details (name, handle, bio, followers, verification status).
  • Media links (high-quality images, videos, thumbnails).
  • Engagement statistics.
• Privacy & Reliability:
  • Uses residential proxies (optional) for better success rates.
  • Built-in retry logic to handle transient failures.
  • No API key required.

📋 Usage

Input Parameters

The Actor accepts a wide range of input parameters to fine-tune your search. Here are the key ones:

Search Configuration

• query: The main search term (keywords, hashtags, etc.).
• usersFromUsers: List of usernames to scrape tweets from (triggers Profile Search if single user).
• numberOfTweets: Maximum number of items to retrieve.

Filters

• mediaTypes: Filter by image, video, media.
• engagementLevel: Presets like low, medium, high, viral.
• timeSince / timeUntil: Date range for tweets (YYYY-MM-DD).
• geoNear: Location to search near (e.g., "New York").

Output Data

The Actor stores results in the default dataset. Each item represents a tweet or user profile.

Example Tweet Output:

{
  "searchType": "post",
  "type": "tweet",
  "query": "python programming",
  "data": {
    "tweet_url": "https://x.com/username/status/123456789",
    "tweet_id": "123456789",
    "author": {
      "fullname": "Python Dev",
      "username": "pythondev",
      "verified": true
    },
    "text": "Just released a new Python library! #python #coding",
    "stats": {
      "likes": 150,
      "retweets": 45,
      "comments": 12
    },
    "media": [
      {
        "type": "image",
        "url": "https://pbs.twimg.com/media/..."
      }
    ]
  }
}

🛠️ Development

This Actor is built with Python using the Apify SDK and Crawlee.

Local Setup

1. Clone the repository.
2. Install dependencies:

   $pip install -r requirements.txt
   

3. Run the Actor:

   $apify run
   

📄 License

This project is licensed under the Apache License 2.0.