Google Shorts Scraper

Discover short-form video content through Google Search, including YouTube Shorts, TikTok videos, Instagram Reels, and other short-video results. Extract titles, URLs, channels, platforms, durations, thumbnails, and short-video indicators.

Features

• Short-Video Search — Search Google for short-form videos using targeted keywords and topics.
• Multi-Platform Discovery — Find short videos from YouTube, TikTok, Instagram, and other supported platforms.
• Video Metadata — Collect titles, video links, channels, platforms, durations, thumbnails, and isShort status.
• Keyword-Based Research — Search multiple topics or niches within a single run.
• Pagination Support — Collect results across multiple Google Search result pages.
• Configurable Limits — Control the number of video results returned per search.
• Structured Output — Export clean video datasets for analysis and reporting.
• Cloud Ready — Runs on Apify with scheduling, proxy support, webhooks, datasets, and API integration.

Use Cases

• Research trending short-form video content.
• Discover viral videos and emerging content trends.
• Analyze short-video strategies across competitors.
• Build searchable short-form video catalogs.
• Monitor niche-specific video topics and creators.
• Collect content sources for market and audience research.

Local Development

⚠️ Important: Do NOT run Google search scrapers on your local network without proper proxy configuration. Doing so will likely result in:

• Immediate IP Blocking: Google blocks IPs that make too many requests.
• CAPTCHAs: You will be forced to solve CAPTCHAs.
• Poor Results: Slow performance and unreliable data.

To run this scraper locally:

1. Install Dependencies:

   $npm install
   

2. Configure Proxy (Required):

   • Get a residential or datacenter proxy service (e.g., Oxylabs, Bright Data, Apify Proxy).
   • Option A (Recommended): Add your proxy configuration to the local input.json:

     {
       "queries": ["your query"],
       "maxItems": 50,
       "proxyConfiguration": {
         "useApifyProxy": true
       }
     }
     

   • Option B: Set environment variables (not recommended for local testing):

     $export APify_USE_APIFY_PROXY=true
     

3. Run the Scraper:

   $npx crawlkit-cli dev --input test-input.json
   

4. Verify Results: Check the storage/datasets/default/OUTPUT.json file for scraped data.

Example Output

{
  "position": 1,
  "title": "Funny Cats Compilation 2024",
  "link": "https://www.youtube.com/shorts/dQw4w9WgXcQ",
  "platform": "YouTube",
  "channel": "Cute Animals",
  "duration": "0:58",
  "isShort": true,
  "thumbnail": "https://i.ytimg.com/vi/dQw4w9WgXcQ/hqdefault.jpg",
  "query": "funny cats",
  "page": 1,
  "scrapedAt": "2026-05-20T10:30:00.000Z"
}