Video-to-Playwright Automation Actor

Transform screen recordings into executable Playwright automation scripts using Google Gemini AI vision capabilities.

🎯 What It Does

This actor analyzes video recordings of user interactions with websites and automatically generates Python Playwright scripts that replicate those actions. Perfect for:

• QA Testing: Record manual tests once, replay them automatically
• Web Scraping: Show the actor how to navigate a site, get a working script
• RPA Automation: Convert manual workflows into automated browser tasks
• Documentation: Generate script documentation from video demos

🚀 How It Works

1. Upload Video: Place your screen recording (mp4, avi, mov, webm) in the videos directory
2. AI Analysis: Google Gemini 2.0 Flash analyzes every frame to identify:
   • Mouse clicks and movements
   • Keyboard inputs and text entries
   • Navigation patterns
   • Element interactions (buttons, forms, links)
3. Script Generation: Creates a complete, production-ready Playwright Python script with:
   • Proper selectors (CSS, text-based, IDs)
   • Wait conditions and timeouts
   • Error handling and screenshots on failure
   • Detailed comments explaining each step
4. Auto-Execution: Optionally runs the generated script to verify it works
5. Storage: Saves scripts to Apify key-value store for later use

📋 Input Parameters

Parameter	Type	Description	Default
videoFile	string	Name of video file in videos directory	test_1.mp4
taskDescription	string	Optional context about the task shown	-
includeScreenshots	boolean	Add screenshot captures to script	false
executeAfter	boolean	Auto-run script after generation	true
headless	boolean	Run browser in headless mode	false
slowMo	integer	Slow motion delay (ms) for debugging	100

🔧 Environment Variables

Set these in the Apify Console:

• GEMINI_API_KEY (required): Your Google Gemini API key - Get one here
• VIDEO_UPLOAD_DIR: Custom directory for videos (default: ./videos)
• MAX_VIDEO_SIZE_MB: Max upload size in MB (default: 100)
• GEMINI_MODEL: Model to use (default: gemini-2.0-flash-exp)

📦 Outputs

Key-Value Store

• generated_script.py: The complete Playwright automation script
• Additional scripts if you request modifications

Dataset (if save_output=true)

• Execution results with stdout/stderr
• Success/failure status
• Return codes

💡 Example Usage

Basic Usage

{
  "videoFile": "login_workflow.mp4",
  "taskDescription": "User logging into dashboard",
  "executeAfter": true
}

Advanced Usage

{
  "videoFile": "checkout_process.mp4",
  "taskDescription": "Complete e-commerce checkout flow",
  "includeScreenshots": true,
  "executeAfter": true,
  "headless": false,
  "slowMo": 500
}

🎥 Video Recording Tips

For best results when recording videos:

1. Clear Actions: Perform actions deliberately with visible mouse movements
2. Wait for Loads: Pause after page loads and before clicking
3. Stable Elements: Interact with elements that have consistent selectors
4. Resolution: Record in 1080p or higher for better element detection
5. Duration: Keep videos under 2-3 minutes (Gemini context limits)
6. Single Task: One clear workflow per video

🏗️ Architecture

┌─────────────────┐
│   Upload Video  │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Gemini Vision  │ ◄── Frame-by-frame analysis
│   AI Analysis   │     Interaction detection
└────────┬────────┘     Element identification
         │
         ▼
┌─────────────────┐
│ Script Generator│ ◄── Playwright template
│  (AI Prompted)  │     Selector optimization
└────────┬────────┘     Error handling
         │
         ▼
┌─────────────────┐
│  Execute Script │ ◄── Browser automation
│   (Optional)    │     Verification
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Save to Store  │ ◄── Key-value storage
│   Return Results│     Dataset output
└─────────────────┘

🛠️ Technical Details

Supported Websites

• Works best with public websites (no login walls for search engines)
• Handles dynamic content (SPAs, AJAX)
• Supports consent dialogs and popups
• Compatible with YouTube, Google, e-commerce sites, etc.

Script Features

Generated scripts include:

• Async/await pattern for modern Playwright
• Proper browser context setup
• Viewport and user-agent configuration
• Timeout management (15s elements, 30s navigation)
• Try/catch error handling
• Screenshot on failure
• Console logging for debugging

Limitations

• Video must be clear and not too fast
• Complex multi-step workflows may need script refinement
• Personalized content (like YouTube homepage) requires search functionality
• Very long videos (>5 min) may hit token limits

🔄 MCP Server Mode

The actor can also run as an MCP (Model Context Protocol) server for interactive use:

Set AUTO_ANALYZE_VIDEO=false to enable MCP mode, then use these tools:

• analyze_video: Generate script from video
• modify_script: Refine script with natural language
• execute_script: Run the automation
• get_script: Retrieve current script
• save_script: Store to key-value store

📚 Example Output

from playwright.async_api import async_playwright
import asyncio

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False)
        page = await browser.new_page()
        
        # Step 1: Navigate to YouTube
        print("Navigating to YouTube...")
        await page.goto('https://www.youtube.com')
        
        # Step 2: Search for video
        print("Searching...")
        search_input = page.locator('input[name="search_query"]')
        await search_input.fill('Minecraft gameplay')
        await page.keyboard.press('Enter')
        await asyncio.sleep(2)
        
        # Step 3: Click first video
        print("Clicking video...")
        video = page.locator('a#video-title').nth(0)
        await video.click()
        
        print("[SUCCESS] Automation completed!")
        await browser.close()

asyncio.run(main())

🐛 Troubleshooting

Script times out finding elements:

• Video content may not be available (e.g., personalized feeds)
• Use search functionality instead of expecting content on homepage
• Increase timeout values in generated script

Unicode/encoding errors:

• Already handled with UTF-8 subprocess environment
• If issues persist, remove emojis from print statements

Video upload fails:

• Check file size (max 100MB by default)
• Ensure video format is supported (mp4, avi, mov, webm)
• Verify video path in VIDEO_UPLOAD_DIR

Generated script doesn't work:

• Use modify_script tool to refine selectors
• Add more wait conditions
• Handle dynamic content with explicit waits

📄 License

MIT

🔗 Links

• Apify Platform
• Playwright Documentation
• Google Gemini API
• MCP Protocol

👥 Support

For issues or questions:

1. Check the execution logs in Apify Console
2. Review generated script for errors
3. Try modifying the script with natural language instructions
4. Ensure your Gemini API key is valid and has quota