# Changelog of YouTube Subtitle & Transcript Scraper (`autofacts/youtube-subtitle-transcript-scraper`) Actor - **URL**: https://apify.com/autofacts/youtube-subtitle-transcript-scraper/changelog.md - **Full Actor documentation**: https://apify.com/autofacts/youtube-subtitle-transcript-scraper.md ## Changelog All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### \[Unreleased] #### Fixed - **Playlist/channel pagination never ran (critical).** `extractVideoIdsFromData` read the next-page token from a top-level `"continuation"` key that YouTube no longer emits anywhere in `ytInitialData` — so the token was always `null`, the pagination loop body never executed once, and collection silently stopped after page 1. Token discovery now walks the payload for any `continuationCommand.token`, which covers both live nestings (`continuationItemRenderer.continuationEndpoint…` on channels, `continuationItemViewModel.continuationCommand.innertubeCommand…` on playlists) and any future re-nesting. - **Channels returned ~7% of their videos.** `resolveChannelVideoIds` keyed off `meta[itemprop="channelId"]`, which YouTube stopped emitting; the uploads-playlist route was therefore never taken and it fell through to a no-pagination HTML sweep. A 448-video channel returned 30. Channel ID resolution now tries the meta tag, the canonical link, `og:url`, `channelMetadataRenderer.externalId`, and a raw HTML regex in turn. - **Video IDs missed on the current grid layout.** Channel and playlist grids now render as `lockupViewModel`, where the ID moved to `contentId`; extraction reads it (gated on `contentType` so playlist and channel lockups aren't mistaken for videos) alongside the classic `*VideoRenderer.videoId` shapes. - **Pagination truncated at a random point (proxy IP churn).** Playlist/channel requests were made on the session-less base proxy URL, so Apify drew a *different* residential IP for every request. A flagged IP answers a continuation with zero items, which reads as end-of-list — so a 448-video channel came back with 90-300 videos, varying run to run. Each pagination sequence now pins one proxy session for the whole chain: instead of taking an independent chance on IP quality 27 times, it takes that chance once. (Measured: rotating the IP per page collapses the same channel to 30 videos; one pinned session returned 448/448 and 801/801 on repeated runs.) Note this is *not* token-IP binding — continuation tokens were verified portable, returning identical items from any exit IP. - **Empty continuation pages read as end-of-list.** When an exit IP is flagged, YouTube answers with zero items — indistinguishable from the last page. Pagination now re-asks such a page on a *fresh* proxy session (up to twice) and adopts that session for the remaining pages if it works; retrying on the same session would only re-ask the IP that just refused. - **Next-page token missed when YouTube moves it.** The token is usually the last entry of `continuationItems`, but is intermittently placed elsewhere in the response; searching only the items array truncated a 448-video channel at a random 90-120. Token lookup now falls back to a whole-body search. - **`llm` output kept `>>` speaker markers.** The `/>>.*?:/g` pattern required a colon, so the bare `>>` markers real caption tracks use survived into "clean" text (`">> THIS IS A LOT OF WATER."`); when a colon *did* appear later in the line, the non-greedy match silently ate every word in between. The speaker name is now optional and length-bounded, and `cleanSegmentText` strips markers too. - **Proxy bypass via impit (critical).** `impit`'s per-request `proxyUrl` option is silently dropped — it is only honored on the `Impit` constructor. Previously every InnerTube and embed-page request was hitting YouTube directly from the actor container's data-center IP, which is what tanked the 7-day success rate to ~11.8%. Proxy is now passed at construction. - **Session ID collisions under concurrency.** `newSession()` used `Date.now() + 4 base36 chars` (~1.6M values) — well into birthday-collision range with the worker pool. Switched to `crypto.randomBytes(8)` (64 bits). - **Player → caption IP split.** Embed-FastPath and WEB-WatchPage previously fetched the player JSON on session A and then the caption file on session B, letting YouTube cross-check the IP mismatch. These paths now inherit the same sessioned proxy as their player fetch. - **Retry storms on dead IPs.** The old two-level retry (`tryMethod` outside, `withRetry` inside) burned up to 3 attempts on the same flagged session before rotating. Collapsed to a single retry layer in `tryMethod`, which rotates session on **every** retry — bot-block OR retryable network error. - **Bot-block detection blind spots.** `isBotBlocked` now matches `HTTP 403/429/503`, empty caption bodies, consent walls, and "could not extract player response" — the realistic Apify-residential blackhole signals, not just `LOGIN_REQUIRED` text matches. - **Insight 500 misclassification.** `classifyErrorStatus` (the status reported to Scraper Insight) didn't recognize the bot-block surfaces `isBotBlocked` already catches — a raw `HTTP 403`/`forbidden`, "could not extract player response", and empty caption bodies — so real bot-blocks were logged as generic `500`s. They now map to `403`, and the `404` "unavailable" matcher was broadened ("video is unavailable", "isn't available", "no longer available"). The `403` check runs first so a bot-block's *fake* "video unavailable" still classifies as `403`. #### Added - **ANDROID and IOS InnerTube clients, now leading the extraction chain.** Measured over 10 videos (caption actually fetched and parsed, not just a player `OK`): ANDROID 10/10 with no visitorData, IOS 9/10, against ANDROID\_VR — the previous primary — at 1/10. Both return caption URLs without `exp=xpe`, so neither needs a PO token. ANDROID needs no `/embed` preflight either, so most videos now resolve on the first request instead of after a preflight plus 1-3 failed attempts. Measured end-to-end on 8 videos (4 of which previously reached the yt-dlp fallback): 26-28s with 0 preflights and 0 yt-dlp invocations, against 51-78s with 8 preflights and a yt-dlp fallback on the old chain. - **Client versions refreshed from yt-dlp at startup.** `clientVersion` staleness is the biggest single cause of silent success-rate decay — the same ANDROID client on the same video returns OK on 21.02.35, LOGIN\_REQUIRED on 20.10.38, and HTTP 400 on anything older, degrading on hard videos first so it rots invisibly. yt-dlp already ships in the image and maintains this table, so `refreshClientVersions()` reads `INNERTUBE_CLIENTS` at boot and re-points every client, rebuilding the User-Agent in step (the native clients embed their version in the UA and YouTube cross-checks the two). Falls back to the compiled-in literals. This immediately corrected three hardcoded versions that were future-dated and invalid. - **Under-collection warning for playlists.** YouTube stops issuing continuation tokens partway through large playlists (measured: uploads playlists stop at \~200 items). That server-side cap cannot be paged past, but the shortfall is no longer silent — when a playlist advertises more videos than were collected, the count of both is logged as a warning. - **Pagination guards.** Hard 50-page cap, stop-on-no-new-IDs, cross-page deduplication, and a jittered 0.5-1s gap between continuation requests. A mid-pagination failure now returns the pages already collected instead of discarding them. `paginate()` takes injectable dependencies so the empty-page recovery path is unit-tested deterministically — a live empty response requires drawing a flagged proxy IP and cannot be forced on demand. - **Proxy egress self-test on startup.** Hits `api.ipify.org` through one session and logs the exit IP, so the next time the proxy gets unplumbed it shows up in the first log line instead of a multi-week Insight regression. - **Cross-worker circuit breaker.** When 5 consecutive videos fail across the worker pool, all workers pause for 30s — prevents the thundering-herd burn through proxy sessions when YouTube changes detection mid-run. - **Optional `apifyProxyCountry` config.** Pin Apify residential exit IPs to a specific country (e.g. `"US"`). YouTube flags residential IPs from low-trust regions ~5–10× more aggressively, so US/EU pinning typically pushes success rate higher. Default is unspecified to avoid silently changing behavior for existing users. - **`test-proxy-rotation.mjs` diagnostic script.** Probes the Apify proxy directly: 5 fresh sessions should return 5 distinct exit IPs; the same session ID called twice should return the same IP. Run via `npm run test:proxy` inside `apify run` for credential injection. - **PO-token provider — the real `exp=xpe` unlock.** The image now bundles the BgUtils PO-token provider (`bgutil-ytdlp-pot-provider`, *script* mode, pinned to `1.3.1` for both the yt-dlp Python plugin and the Node server build), and the yt-dlp fallback routes minting through it via `--extractor-args youtubepot-bgutilscript:server_home=…`. yt-dlp on its own cannot mint the Proof-of-Origin tokens that gate `exp=xpe` caption URLs and the "Sign in to confirm you're not a bot" wall — this is the only path that actually unblocks those videos. The provider's BotGuard runner is built against node-canvas, so the Alpine image carries the Cairo/Pango/JPEG/giflib stack at runtime (toolchain + headers are build-only and removed). Server location is overridable via the `BGUTIL_SERVER_HOME` env var. - **PO-token provider self-test on startup.** Runs `generate_once.js --version` at init and logs `bgutil ` (or a clear warning), so a broken provider surfaces in the first log lines instead of only when a hard video reaches the fallback — mirrors the proxy egress self-test. #### Changed - `/embed` preflight demoted to step 3 and its purpose corrected in the source. It was documented as "the single highest-success-rate endpoint for transcript scraping" with a caption fast-path, but returned **zero caption tracks on every video tested** — the fast-path never fires. Its actual contribution is the visitorData that lifts ANDROID\_VR from 1/10 to 9/10, so it now runs only after ANDROID and IOS have both failed. - Channel resolution now paginates the `/videos` tab instead of the uploads playlist. The uploads playlist pages 100 videos per request vs ~30, but stops issuing tokens at ~200 items; the `/videos` tab reaches the full list (verified against yt-dlp: 448/448 and 801/801). The uploads playlist is kept as a fallback for when the tab yields nothing. - InnerTube `browse` calls dropped the hardcoded `?key=AIzaSy…` API key — verified to return identical results without it, and one less credential to be rotated out from under us. - `got-scraping` is now a declared dependency. `playlist.ts` always imported it, but it resolved only transitively through `apify` and would have broken on any dependency-tree change. - Per-request timeout bumped from 20 s to 35 s — residential-proxy realistic. - Inter-method delay bumped from 500 ms to 1500–2500 ms (jittered) and retry cooldown from 500–1000 ms to 2000–4000 ms (jittered). Shorter cooldowns kept pulling another IP from the same /24 that YouTube just flagged. - Debug logs now include `[session=]` on every attempt and failure for correlation against Apify's session-usage dashboard. - **Cookie path pins one exit IP per video (sticky session).** When cookies are supplied, every method and retry for a video reuses a single proxy session instead of rotating a fresh IP per attempt. Cookies carry a stable account identity, and fanning it across many residential IPs in seconds is impossible travel that flags the account. Without cookies, the rotate-per-attempt anonymous strategy is unchanged. ### \[0.2.0] - 2026-04-19 - Initial release: YouTube subtitle & transcript scraper with embed preflight, caption fast-path, multi-client InnerTube fallback (ANDROID\_VR, TVHTML5\_EMBED, WEB\_EMBEDDED, MWEB, WEB), and yt-dlp last-resort fallback.