Twitter Video Downloader

A powerful Apify Actor that downloads videos from Twitter/X posts with support for both direct post URLs and username-based scraping. Built for reliable video downloads and authenticated profile scraping.

A template for web scraping data from a single web page in Python. The URL of the web page is passed in via input, which is defined by the input schema. The template uses the HTTPX to get the HTML of the page and the Beautiful Soup to parse the data from it. The data are then stored in a dataset where you can easily access them.

The scraped data in this template are page headings but you can easily edit the code to scrape whatever you want from the page.

Included features

• Apify SDK for Python - a toolkit for building Apify Actors and scrapers in Python
• Input schema - define and easily validate a schema for your Actor's input
• Request queue - queues into which you can put the URLs you want to scrape
• Dataset - store structured data where each object stored has the same attributes
• HTTPX - library for making asynchronous HTTP requests in Python
• Beautiful Soup - library for pulling data out of HTML and XML files

How It Works

Input Modes

1. Direct Post URLs

{
  "postUrls": [
    "https://x.com/username/status/123456789"
  ],
  "videoQuality": "best"
}

2. Username-Based Scraping

{
  "usernames": ["elonmusk", "NASA"],
  "maxPostsPerUser": 20,
  "videoQuality": "720p"
}

Workflow

1. Input Processing - Actor accepts either post URLs or usernames
2. Profile Scraping (username mode) - Playwright browser collects post URLs from user profiles
3. Video Download - downloads videos with selected quality
4. Metadata Extraction - OpenCV extracts video specifications (resolution, FPS, codec, duration, bitrate)
5. Cloud Upload - Videos uploaded to Apify key-value store
6. Dataset Output - Structured data saved with download links

Output

Each downloaded video includes:

{
  "tweet_url": "https://x.com/username/status/123456789",
  "id": "123456789",
  "title": "Tweet title",
  "uploader": "username",
  "filename": "123456789.mp4",
  "resolution": "1920x1080",
  "download_status": "finished",
  "download_url": "https://api.apify.com/v2/key-value-stores/.../records/video_123456789.mp4",
  "storage_key": "video_123456789.mp4",
  "downloaded_at": "2025-11-13T00:00:00.000000",
  "video_meta": {
    "width": 1920,
    "height": 1080,
    "fps": 30.0,
    "duration": 45.23,
    "video_codec": "avc1",
    "ext": "mp4",
    "filesize_bytes": 28640256,
    "total_bitrate_kbps": 5064.12,
    "aspect_ratio": 1.78
  }
}

Input Parameters

Parameter	Type	Required	Default	Description
postUrls	Array	No	[]	List of Twitter/X post URLs to download
usernames	Array	No	[]	List of usernames to scrape videos from (without @)
maxPostsPerUser	Integer	No	20	Maximum posts to scrape per username (1-100)
videoQuality	String	No	best	Video quality: best, 720p, 480p, worst
browserCookies	String	No	-	Browser cookies in JSON format (for profile scraping)
minDelayBetweenRequests	Integer	No	2	Minimum delay between requests (1-30 seconds)
maxDelayBetweenRequests	Integer	No	5	Maximum delay between requests (1-60 seconds)
humanizeBehavior	Boolean	No	true	Enable human-like mouse movements and scrolling

Use Cases

1. Content Archival

Archive videos from your favorite Twitter accounts before they're deleted:

{
  "usernames": ["NASA", "SpaceX"],
  "maxPostsPerUser": 50,
  "videoQuality": "best"
}

2. Research & Analysis

Collect videos for social media research:

{
  "usernames": ["researcher1", "researcher2"],
  "maxPostsPerUser": 100,
  "videoQuality": "720p"
}

3. Specific Video Download

Download individual videos:

{
  "postUrls": [
    "https://x.com/username/status/123456",
    "https://x.com/username/status/789012"
  ],
  "videoQuality": "best"
}

Cookie Configuration

Why Use Cookies?

Cookies help avoid rate limits and access protected content. The actor uses a dual cookie system:

1. Browser Cookies (browserCookies parameter) - For Playwright profile scraping
2. cookies.txt File - For reliable video downloads

Getting Cookies

Option 1: Browser Extension

1. Install "Get cookies.txt" extension for your browser
2. Navigate to x.com and log in
3. Click extension icon to export cookies
4. Save as cookies.txt in actor root directory

Option 2: Manual Export

Use browser DevTools to export cookies in JSON format and paste into browserCookies input field.

Default Cookies

The actor includes default browser cookies for basic profile scraping. For heavy usage, provide your own authenticated cookies.

Anti-Detection Features

• Human-like delays - Random delays between 2-5 seconds (configurable)
• Mouse movements - Simulated cursor movements during scraping
• Scroll behavior - Natural scrolling patterns
• Firefox browser - Less detectable than Chrome for automation
• Custom user agents - Realistic browser fingerprints
• Rate limit protection - Automatic detection of blocks and warnings

Local Development

Prerequisites

• Python 3.12+
• Apify CLI (npm install -g apify-cli)

Setup

1. Clone the repository:

$cd Twitter/twitter-video-downloader-cli

2. Install dependencies:

pip install -r requirements.txt
playwright install firefox

3. Create input file .actor/INPUT.json:

{
  "postUrls": ["https://x.com/username/status/123456"],
  "videoQuality": "best"
}

4. (Optional) Add cookies.txt for authenticated downloads

5. Run locally:

$apify run

Testing

The actor creates local storage in ./storage/:

• datasets/default/ - Output data with metadata
• key_value_stores/default/ - Input configuration
• Downloaded videos are uploaded to cloud storage

Deployment

Deploy to Apify Platform

1. Login to Apify:

$apify login

2. Push to Apify:

$apify push

3. Configure Secrets (optional):
   • Add cookies.txt as a secret file in Actor settings
   • Set environment variables for sensitive data

Deploy from GitHub

1. Go to Apify Console
2. Click "Link Git Repository"
3. Connect your GitHub repository
4. Configure build settings:
   • Build tag: latest
   • Dockerfile: ./Dockerfile

Performance & Limits

Speed

• Profile scraping: ~2-5 seconds per username
• Video download: 5-30 seconds per video (depends on quality/size)
• Metadata extraction: < 100ms per video (OpenCV)

Rate Limits

• Default delays: 2-5 seconds between requests
• Recommended max: 100 posts per run without authentication
• With cookies: 500+ posts per run (monitor for blocks)

Quality vs. File Size

Quality	Resolution	Typical Size	Use Case
best	1080p-4K	20-100 MB	Archival, high quality
720p	1280x720	10-40 MB	Balanced quality/size
480p	854x480	5-15 MB	Fast downloads
worst	360p-240p	2-8 MB	Bandwidth limited

Troubleshooting

"No video found" Error

• Tweet may not contain a video (only images/GIFs)
• Video may be age-restricted or geo-blocked
• Try adding cookies.txt for authentication

Rate Limited / Blocked

• Increase minDelayBetweenRequests and maxDelayBetweenRequests
• Enable humanizeBehavior
• Add authenticated browser cookies
• Reduce maxPostsPerUser

Video Quality Issues

• our actor selects best available format within quality constraint
• Some tweets only have lower quality versions
• Check video_meta in output for actual resolution

Missing Metadata (FPS, Duration)

• Some HLS streams don't expose all metadata upfront
• Metadata is extracted post-download via OpenCV
• Check logs for extraction errors

API Integration

Fetch Results

from apify_client import ApifyClient

client = ApifyClient('YOUR_API_TOKEN')

# Start actor run
run = client.actor('YOUR_ACTOR_ID').call(run_input={
    "usernames": ["NASA"],
    "maxPostsPerUser": 10,
    "videoQuality": "best"
})

# Fetch dataset items
dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items

for item in dataset_items:
    print(f"Video: {item['download_url']}")

Download Videos

Videos are stored in key-value store. Use the download_url from output:

$curl -o video.mp4 "https://api.apify.com/v2/key-value-stores/{STORE_ID}/records/{KEY}"

Limitations

• Only downloads videos (not images or GIFs)
• Requires video to be embedded in tweet (not external links)
• Private/protected accounts require authenticated cookies
• Age-restricted content requires authentication
• Twitter's rate limits apply (use delays and cookies)

Support

• Apify Discord Community
• Apify Documentation

License

This project is for educational and personal use. Respect Twitter's Terms of Service and robots.txt. Always ensure you have rights to download and use the content.

---

Built with ❤️ by CrawlerBros