Youtube Comments Scraper

This changelog summarizes all changes of the Youtube actors provided by the Streamers organization. If some change affects only a subset of actors, they will be listed in parentheses.

2024-12-11

Fixes

• Channel username is now correctly extracted for shorts (streamers/youtube-scraper, streamers/youtube-channel-scraper, streamers/youtube-shorts-scraper)
• Fixed extraction of channel URLs and channel IDs (streamers/youtube-scraper, streamers/youtube-channel-scraper, streamers/youtube-shorts-scraper)

2024-12-09

Features

• Can now scrape comments on shorts (streamers/youtube-comments-scraper)

2024-12-04

BREAKING

• If values in any of the maxResults* input properties are omitted, they are initialized to zeroes to avoid unintentional infinite scraping. If you actually want to scrape all the videos, make sure to initialize these fields to e.g. 9999999 Features
• Channel usernames are now extracted (streamers/youtube-scraper, streamers/youtube-channel-scraper)

2024-11-25

Changes

• Input for search terms is now a stringList (a list of strings) instead of a regular string (streamers/youtube-scraper) Features
• Added input output property for search items (streamers/youtube-scraper)

2024-11-21

Fixes

• Video count is extracted correctly for channels

2024-11-18

BREAKING

• All actors now require at least 1GB of run memory Fixes
• A dataset result with an error note is pushed for blocked channels (or any channels that are inaccessible for some reason) instead of failing

2024-10-29

Fixes

• Comments are now again collected if you select sorting order (streamers/youtube-comments-scraper)

2024-10-01

Features

• The URL to the source media file is now extracted for regular videos (formats output property). Although it usually only is in 360p resolution, it combines both audio and video, and we plan to extract sources of better resolutions in the future, as well as sources for shorts. The URL has an expiration time of several hours

2024-09-20

Fixes

• Exact upload date is extracted again

2024-09-10

Fixes

• Shorts are now extracted for channels

2024-08-29

Features

• Added scraping functionality for hashtags (streamers/youtube-scraper)
  • Scrape all video type hashtags https://www.youtube.com/hashtag/apify
  • Scrape shorts only hashtags https://www.youtube.com/hashtag/apify/shorts
• We will soon release a standalone scraper for hashtags :)

Fixes

• Sub count is now extracted correctly

2024-08-21

Fixes

• Fixed "Request batch insert failed" error when scraping search and channels (streamers/youtube-scraper)

2024-08-19

Features

• You can now select comment sorting option in the input (streamers/youtube-comments-scraper)

2024-08-08

Fixes

• Fixed cases when scraper would get stuck in a row of errors when date filter was reached

2024-07-29

Fixes

• Detailed channel information is again extracted correctly when you provide channel URLs

2024-07-18

Fixes

• Fixed date extraction for recently finished premieres

2024-07-17

Features

• Hashtag info has been added to the output (the once in the super-title, above description; hashtags from description are separately present in descriptionLinks) (streamers/youtube-scraper)
• isAgeRestricted channel attribute is now present in output items (except for direct video URLs) Fixes/BREAKING
• Scraper now again can bypass YT blocking. However, the following attributes are currently missing:
  • thumbnailUrl and duration if the video was scraped via a direct URL;
  • duration for all cases with shorts;
  • isMonetized in all cases
• Correctly handles age-restricted channels by pushing just their channel info

2024-07-03

Fixes

• Scraper now uses Datacenter proxies again, which should make it faster

2024-07-02

Features

• Added published time with publishedTimeText property (streamers/youtube-comments)

2024-07-01

Fixes

• Date filters for channels now work correctly

2024-06-29

Fixes

• Restored functionality of the scraper
• RESIDENTIAL proxies are now used

2024-05-14

Fixes

• type field designating whether the video is a short, stream or a regular video has been added for missing cases
• Shorts with turned off comments are now correctly scraped

2024-05-09

Fixes

• Channel avatar, banner and isVerified fixed in several rare cases (*)

2024-04-30

Fixes

• Fixed comments scraping and crawling issues (streamers/youtube-comments-scraper)

2024-04-17

Fixes

• Approximate date filtering is now supported (streamers/youtube-channel-scraper)

2024-04-11

Features

• If scraper doesn't collect any results (e.g. because the channel or search has no videos) it will push an "empty" item with input, URL and note about the reason why no results were collected.

2024-04-09

Features

• shorts sorting/filtering by date has also been added to the input (streamers/youtube-shorts-scraper)

2024-04-05

Features

• added isChannelVerified, channelAvatarUrl, channelBannerUrl fields to output when scraping channels

2024-04-02

Features

• added "Any" subtitle language option to download all available subtitles

2024-03-06

Features

• in addition to channel username, you can now also use channel link for the shorts scraper (streamers/youtube-shorts-scraper)

2024-02-29

Features

• added support for shorts URLs
• added shorts text (description), commentsCount and likes
• added channelId to videos scraped from channel pages and from video detail pages, i.e. channelId won't be present when using search_query URL in streamers/youtube-channel-scraper.

2024-02-13

Fixes

• BREAKING CHANGE: store total channel's views (channelTotalViews) as a number, not a string

2024-02-12

Features

• Added type and replyToCid to comments (streamers/youtube-comments-scraper)

2024-02-06

Fixes

• Fixed scraping voteCount in comments (streamers/youtube-comments-scraper)

2024-01-18

Fixes

• Fixed scraping shorts when using searchKeywords

2024-01-16

Fixes

• inputChannelUrl now contains unmodified URL from the input

Features

• Added channelDescriptionLinks

2023-12-19

Fixes

• Added missing thumbnailUrls field (streamers/youtube-channel)
• Actor should now get correct number of results with respect to "Maximum search results" (streamers/youtube-scraper)

2023-12-07

Fixes

• video likes extraction
• apply date filters also to streams and shorts

2023-11-21

Fixes

• Fixed pagination when using search keywords

2023-11-16

Features

• add new comment fields: replyCount, voteCount, authorIsChannelOwner, hasCreatorHeart (streamers/youtube-comments)

2023-11-11

Fixes

• Fixed pagination scraping - actor now scrapes more than just one page

2023-11-08

Fixes

• Fixed channel info extraction after Youtube changed API

2023-11-01

Features

• Scrape shorts' date and duration

2023-09-27

Fixes

• Fix numberofubscribers extraction (streamers/youtube-main)

Features

• Added isMonetized field for video detail (streamers/youtube-main)

2023-09-12

Features

• Added basic sorting/filtering for channel videos. More might be coming in the future (streamers/youtube-main, streamers/youtube-channel)

2023-08-28

Fixes

• For some videos not all comments used to get scraped, this is now fixed (streamers/youtube-comments)

2023-08-17

Fixes

• Proxy is again used

Features

• You can now apply search filters and sorting in the input (streamers/youtube-main)

2023-08-14

Fixes

• The scraper now again respects limits, which were broken in the previous release (*).

Features

• The search workflow now can also pick up Shorts and Live videos from the corresponding tabs. Set the appropriate limits for each type of videos.

2023-07-31

Input changes

• You are allowed to pass 0 to the max limits for shorts/streams (bernardo/youtube-scraper).

2023-07-25

Fixes

• The scraper now correctly extracts comment count for big videos

2023-07-24

Fixes

• The scraper can now correctly scrape videos without a description

2023-07-07

Features

• Now videos have fromYTUrl and inputChannelUrl fields in the output. The first one tells on what page the video was scraped, the other one will point to the channel url as specified in the input (it may differ from channelUrl, although both will lead to the same channel).

2023-07-06

Features

• Autogenerated channels parsed as "recent" tab

2023-07-04

Fixes

• Subs are now pushed to dataset items, as they used to be

2023-06-22

Fixes

• If a channel doesn't exist, the scraper can now detect it, and will not retry to scrape this channel

2023-06-20

Features

• You can now select in what format to save subtitles: plaintext, vtt, srt or xml

Fixes

• Now the scraper better handles some subtitle locales (for some of them, it would often not download subtitles, because it couldn't match fr with fr-FR, for example)

2023-06-02

Features

• You can now input playlist URLs (in the format of https://www.youtube.com/playlist?list=PLObrtcm1Kw6PmbXg8bmfJN-o2Hgx8sidf) and scrape all videos from it.

Fixes

• You can now submit URLs in the youtu.be/id format. In addition, if you submit some incorrect URLs, they will just be skipped, without exiting the scraper as before.

2023-06-01 (0.0.107)

BREAKING CHANGES

• Removed dislikes field from the output as they are no longer publicly available.
• Removed details field from the output, which was a full HTML version of the description. Use text and descriptionLinks instead.

Features

• Added descriptionLinks field to the output, which contains all links found in the description. Some of them would not be extracted by the text field alone.

Changes

• The scraper is now significantly faster (and thus cheaper) because it no longer requires a full browser interaction.
• extendOutputFunction and extendScraperFunction are deprecated. They will still be supported and we will reach out to users that regularly use it before we completely remove them.

Fixes

• Scrolling through videos can now be restored from any point which makes the scraper much more reliable

2023-04-25

Fixes

• Video duration is now correctly extracted
• Description is now correctly extracted

2023-03-29

Update

• Added new fields to the output in case of processing channelUrl: { "channelTotalVideos": 3200, "channelDescription": "Learn how to speak English with the BBC...", "channelLocation": "United Kingdom", "channelJoinedDate": "Jun 17, 2008", "channelTotalViews": "261,770,375", }

2023-03-29

Feautures

• Added "saveStreams" feature.

2023-02-22

Features

• Added thumbnailUrl to video item output

2023-01-13

Fixes

• Extract the title text only without html
• Extract the description full urls

2022-11-30

Feautures

• Added "saveShorts" feature.

2022-07-20

Fixes

• Correctly handle videos with comments turned off.
• Add commentsTurnedOff to output.

2022-06-10

Fixes:

• Channel page without /watch selector

2021-09-15

Features

• Add possibility to scrape video comments. See maxComments input field.

2021-06-16 Features

• Revamped subtitles downloading - added possibility to download all available subtitles (availability defined by languages) and to prefer automatically generated subtitles before the user generated ones.

2021-06-14 Features:

• Add subtitle type to output (extendedOutputFunction). Note: You must set downloadSubtitles variable to true for this feature to take effect.

2021-06-11 Features:

• Subtitles are now downloadable (saved to KeyValueStore as videoID_languageCode)

2021-05-21 Features:

• Update SDK

Fixes

• Random zero results when searching
• Click consent dialog

2021-04-14 Fixes

• Fixed changed selector that completely prevented the scrape

2021-03-21 Features:

• Updated SDK version for session pool changes
• Add handlePageTimeoutSecs parameter to INPUT_SCHEMA

2021-03-15 Fixes:

• Fixed selector causing no data scraped
• Removed stealth causing issues with new layout

2020-09-27

• Increased waiting timeouts to better handle concurrency
• Added saving screenshots on errors
• Better handling of Captchas, a page is automatically retried and the browser is restarted with a new proxy
• verboseLog is off by default
• Added info how many videos were enqueued and overall better logging