Actor picture

YouTube Scraper

bernardo/youtube-scraper

Scrape YouTube videos with our alternative YouTube API. Extract and download channel name, likes, number of views, and number of subscribers. Scrape by keyword or URL. Limit your searches with human-friendly date formats. Download your data as HTML table, JSON, CSV, Excel, XML, and RSS feed.

Youtube Scraper

Youtube Scraper powered by Apify's Puppeteer crawler

Youtube has an API that gives access to search data as well as detailed data on each video. However, it requires you to log in and imposes quota limits on each API endpoint.

That is why Youtube Scraper was created and implemented as an actor to run on the Apify platform. This scraper is open-source and you can easily run it locally or on your system. Contributions are welcome.

Features of Youtube Scraper:

  • Scrape videos by specifying a search keyword(s) or URLs to get video details:
  • Limit the videos returned by upload date by entering a date in an easy-read format e.g "2 weeks ago"
  • Limit the number of videos returned for each search or channel (focusing on the top videos)

Table of contents

Input parameters

The input of this actor should be JSON specifying what to search for on Youtube. If this actor is run on the Apify platform a user-friendly graphical interface will be provided for you to configure the scraper before running it. This actor recognizes the following input fields:

Field Type Description
searchKeywords String (optional) Query to search Youtube for
maxResults Integer (optional) How many videos should be loaded from each search or channel, default is 50
postsFromDate String (optional) How far back in history to go, default is "5 years ago". You can also use minutes,hours,days,weeks and months
startUrls String (optional) Starting Youtube URLs, you can provide search, channel or videos urls
proxyConfiguration Object Proxy configuration
verboseLog Boolean Whether to turn on verbose logging

This solution requires the use of Proxy servers, either your own proxy servers or you can use Apify Proxy.

Youtube scraper Input example

{
    "searchKeywords": "Terminator dark fate",
    "maxResults": 30,
    "postsFromDate": "2 weeks ago",
    "startUrls": [{
        "url": "https://www.youtube.com/channel/UC8w/videos" // channel videos
    }, {
        "url": "https://www.youtube.com/results?search_query=finances" // search queries
    }, {
        "url": "https://www.youtube.com/watch?v=kJQP7kiw5Fk" // videos
    }],
    "proxyConfiguration": {
        "useApifyProxy": true
    },
    "verboseLog": false
}

During the run

During the run, the actor will output messages letting you know what it is doing and which youtube URL is being processed.

If an error occurs there will be a detail error log in the run console as well as in the output dataset.

Scraper output

As the actor runs, the actor stores results into a dataset. Each Youtube video becomes a separate item in the dataset (example below).

The actor converts Youtube data into a form that can be compared and analyzed as exemplified in the table below:

# Likes Output
1.2M 1200000
1.65K 1650
1.65M 1650000

The output can then be easily manipulated in any language (Python, PHP, Node JS/NPM).

See the Apify API reference to learn more about getting results from this Youtube actor.

Here is a sample of the output (long lines shortened):

{
  "title": "Terminator: Dark Fate - Official Trailer (2019) - Paramount Pictures",
  "id": "oxy8udgWRmo",
  "url": "https://www.youtube.com/watch?v=oxy8udgWRmo",
  "viewCount": 15432,
  "date": "2019-08-29T00:00:00+00:00",
  "likes": 121000,
  "dislikes": 23000,
  "channelName": "Paramount Pictures",
  "channelUrl": "https://www.youtube.com/channel/UCF9imwPMSGz4Vq1NiTWCC7g",
  "numberOfSubscribers": 1660000,
  "details": "Welcome to the day after <a class=\"yt-simple-endpoint style-sco..."
}

Planned features

  • Scraping comments on a video
  • Scraping channel details

Changelog

Changes related to the new versions are listed in the CHANGELOG.md file.

Notes for developers

Typical usage on Apify platform using 4096MB for memory is shown below:

Resource Average Max
Memory 480.3 MB 1.1 GB
CPU 53% 140%

This actor manipulates the mouse and keyboard like a real user would.

It uses xPaths to find DOM elements; they are all stored in one file for easy update.

All xPath variables and functions end in 'Xp'.

The logic of the actor makes use of Youtube's own date filters because:

  • Youtube does not by default show videos in chronological order. The order of videos is related to the number of likes, subscribers, etc.
  • There is no way to give Youtube an exact cutoff date (unless you use Youtube's developer API)
  • Youtube has a separate filter that toggles sorting by date

So when a user requests videos from "5 days ago", we apply Youtube's "This week" filter as well as the "Sort by Upload date" filter.

  • Updated
  • Last run
  • Used4883 times
  • Used by736 users