Youtube Scraper

This changelog summarizes all changes of the Youtube actors provided by the Streamers organization. If some change affects only a subset of actors, they will be listed in parentheses.

2024-04-11

Features

• If scraper doesn't collect any results (e.g. because the channel or search has no videos) it will push an "empty" item with input, URL and note about the reason why no results were collected.

2024-04-09

Features

• shorts sorting/filtering by date has also been added to the input (streamers/youtube-shorts-scraper)

2024-04-05

Features

• added isChannelVerified, channelAvatarUrl, channelBannerUrl fields to output when scraping channels

2024-04-02

Features

• added "Any" subtitle language option to download all available subtitles

2024-03-06

Features

• in addition to channel username, you can now also use channel link for the shorts scraper (streamers/youtube-shorts-scraper)

2024-02-29

Features

• added support for shorts URLs
• added shorts text (description), commentsCount and likes
• added channelId to videos scraped from channel pages and from video detail pages, i.e. channelId won't be present when using search_query URL in streamers/youtube-channel-scraper.

2024-02-13

Fixes

• BREAKING CHANGE: store total channel's views (channelTotalViews) as a number, not a string

2024-02-12

Features

• Added type and replyToCid to comments (streamers/youtube-comments-scraper)

2024-02-06

Fixes

• Fixed scraping voteCount in comments (streamers/youtube-comments-scraper)

2024-01-18

Fixes

• Fixed scraping shorts when using searchKeywords

2024-01-16

Fixes

• inputChannelUrl now contains unmodified URL from the input

Features

• Added channelDescriptionLinks

2023-12-19

Fixes

• Added missing thumbnailUrls field (streamers/youtube-channel)
• Actor should now get correct number of results with respect to "Maximum search results" (streamers/youtube-scraper)

2023-12-07

Fixes

• video likes extraction
• apply date filters also to streams and shorts

2023-11-21

Fixes

• Fixed pagination when using search keywords

2023-11-16

Features

• add new comment fields: replyCount, voteCount, authorIsChannelOwner, hasCreatorHeart (streamers/youtube-comments)

2023-11-11

Fixes

• Fixed pagination scraping - actor now scrapes more than just one page

2023-11-08

Fixes

• Fixed channel info extraction after Youtube changed API

2023-11-01

Features

• Scrape shorts' date and duration

2023-09-27

Fixes

• Fix numberofubscribers extraction (streamers/youtube-main)

Features

• Added isMonetized field for video detail (streamers/youtube-main)

2023-09-12

Features

• Added basic sorting/filtering for channel videos. More might be coming in the future (streamers/youtube-main, streamers/youtube-channel)

2023-08-28

Fixes

• For some videos not all comments used to get scraped, this is now fixed (streamers/youtube-comments)

2023-08-17

Fixes

• Proxy is again used

Features

• You can now apply search filters and sorting in the input (streamers/youtube-main)

2023-08-14

Fixes

• The scraper now again respects limits, which were broken in the previous release (*).

Features

• The search workflow now can also pick up Shorts and Live videos from the corresponding tabs. Set the appropriate limits for each type of videos.

2023-07-31

Input changes

• You are allowed to pass 0 to the max limits for shorts/streams (bernardo/youtube-scraper).

2023-07-25

Fixes

• The scraper now correctly extracts comment count for big videos

2023-07-24

Fixes

• The scraper can now correctly scrape videos without a description

2023-07-07

Features

• Now videos have fromYTUrl and inputChannelUrl fields in the output. The first one tells on what page the video was scraped, the other one will point to the channel url as specified in the input (it may differ from channelUrl, although both will lead to the same channel).

2023-07-06

Features

• Autogenerated channels parsed as "recent" tab

2023-07-04

Fixes

• Subs are now pushed to dataset items, as they used to be

2023-06-22

Fixes

• If a channel doesn't exist, the scraper can now detect it, and will not retry to scrape this channel

2023-06-20

Features

• You can now select in what format to save subtitles: plaintext, vtt, srt or xml

Fixes

• Now the scraper better handles some subtitle locales (for some of them, it would often not download subtitles, because it couldn't match fr with fr-FR, for example)

2023-06-02

Features

• You can now input playlist URLs (in the format of https://www.youtube.com/playlist?list=PLObrtcm1Kw6PmbXg8bmfJN-o2Hgx8sidf) and scrape all videos from it.

Fixes

• You can now submit URLs in the youtu.be/id format. In addition, if you submit some incorrect URLs, they will just be skipped, without exiting the scraper as before.

2023-06-01 (0.0.107)

BREAKING CHANGES

• Removed dislikes field from the output as they are no longer publicly available.
• Removed details field from the output, which was a full HTML version of the description. Use text and descriptionLinks instead.

Features

• Added descriptionLinks field to the output, which contains all links found in the description. Some of them would not be extracted by the text field alone.

Changes

• The scraper is now significantly faster (and thus cheaper) because it no longer requires a full browser interaction.
• extendOutputFunction and extendScraperFunction are deprecated. They will still be supported and we will reach out to users that regularly use it before we completely remove them.

Fixes

• Scrolling through videos can now be restored from any point which makes the scraper much more reliable

2023-04-25

Fixes

• Video duration is now correctly extracted
• Description is now correctly extracted

2023-03-29

Update

• Added new fields to the output in case of processing channelUrl: { "channelTotalVideos": 3200, "channelDescription": "Learn how to speak English with the BBC...", "channelLocation": "United Kingdom", "channelJoinedDate": "Jun 17, 2008", "channelTotalViews": "261,770,375", }

2023-03-29

Feautures

• Added "saveStreams" feature.

2023-02-22

Features

• Added thumbnailUrl to video item output

2023-01-13

Fixes

• Extract the title text only without html
• Extract the description full urls

2022-11-30

Feautures

• Added "saveShorts" feature.

2022-07-20

Fixes

• Correctly handle videos with comments turned off.
• Add commentsTurnedOff to output.

2022-06-10

Fixes:

• Channel page without /watch selector

2021-09-15

Features

• Add possibility to scrape video comments. See maxComments input field.

2021-06-16 Features

• Revamped subtitles downloading - added possibility to download all available subtitles (availability defined by languages) and to prefer automatically generated subtitles before the user generated ones.

2021-06-14 Features:

• Add subtitle type to output (extendedOutputFunction). Note: You must set downloadSubtitles variable to true for this feature to take effect.

2021-06-11 Features:

• Subtitles are now downloadable (saved to KeyValueStore as videoID_languageCode)

2021-05-21 Features:

• Update SDK

Fixes

• Random zero results when searching
• Click consent dialog

2021-04-14 Fixes

• Fixed changed selector that completely prevented the scrape

2021-03-21 Features:

• Updated SDK version for session pool changes
• Add handlePageTimeoutSecs parameter to INPUT_SCHEMA

2021-03-15 Fixes:

• Fixed selector causing no data scraped
• Removed stealth causing issues with new layout

2020-09-27

• Increased waiting timeouts to better handle concurrency
• Added saving screenshots on errors
• Better handling of Captchas, a page is automatically retried and the browser is restarted with a new proxy
• verboseLog is off by default
• Added info how many videos were enqueued and overall better logging