Playlist/channel pagination never ran (critical).extractVideoIdsFromData
read the next-page token from a top-level "continuation" key that YouTube no
longer emits anywhere in ytInitialData — so the token was always null, the
pagination loop body never executed once, and collection silently stopped after
page 1. Token discovery now walks the payload for any continuationCommand.token,
which covers both live nestings (continuationItemRenderer.continuationEndpoint…
on channels, continuationItemViewModel.continuationCommand.innertubeCommand…
on playlists) and any future re-nesting.
Channels returned ~7% of their videos.resolveChannelVideoIds keyed off
meta[itemprop="channelId"], which YouTube stopped emitting; the uploads-playlist
route was therefore never taken and it fell through to a no-pagination HTML sweep.
A 448-video channel returned 30. Channel ID resolution now tries the meta tag,
the canonical link, og:url, channelMetadataRenderer.externalId, and a raw
HTML regex in turn.
Video IDs missed on the current grid layout. Channel and playlist grids now
render as lockupViewModel, where the ID moved to contentId; extraction reads
it (gated on contentType so playlist and channel lockups aren't mistaken for
videos) alongside the classic *VideoRenderer.videoId shapes.
Pagination truncated at a random point (proxy IP churn). Playlist/channel
requests were made on the session-less base proxy URL, so Apify drew a
different residential IP for every request. A flagged IP answers a
continuation with zero items, which reads as end-of-list — so a 448-video
channel came back with 90-300 videos, varying run to run. Each pagination
sequence now pins one proxy session for the whole chain: instead of taking an
independent chance on IP quality 27 times, it takes that chance once.
(Measured: rotating the IP per page collapses the same channel to 30 videos;
one pinned session returned 448/448 and 801/801 on repeated runs.)
Note this is not token-IP binding — continuation tokens were verified
portable, returning identical items from any exit IP.
Empty continuation pages read as end-of-list. When an exit IP is flagged,
YouTube answers with zero items — indistinguishable from the last page.
Pagination now re-asks such a page on a fresh proxy session (up to twice) and
adopts that session for the remaining pages if it works; retrying on the same
session would only re-ask the IP that just refused.
Next-page token missed when YouTube moves it. The token is usually the last
entry of continuationItems, but is intermittently placed elsewhere in the
response; searching only the items array truncated a 448-video channel at a
random 90-120. Token lookup now falls back to a whole-body search.
llm output kept >> speaker markers. The />>.*?:/g pattern required a
colon, so the bare >> markers real caption tracks use survived into "clean"
text (">> THIS IS A LOT OF WATER."); when a colon did appear later in the
line, the non-greedy match silently ate every word in between. The speaker name
is now optional and length-bounded, and cleanSegmentText strips markers too.
Proxy bypass via impit (critical).impit's per-request proxyUrl option
is silently dropped — it is only honored on the Impit constructor.
Previously every InnerTube and embed-page request was hitting YouTube directly
from the actor container's data-center IP, which is what tanked the 7-day
success rate to ~11.8%. Proxy is now passed at construction.
Session ID collisions under concurrency.newSession() used
Date.now() + 4 base36 chars (~1.6M values) — well into birthday-collision
range with the worker pool. Switched to crypto.randomBytes(8) (64 bits).
Player → caption IP split. Embed-FastPath and WEB-WatchPage previously
fetched the player JSON on session A and then the caption file on session B,
letting YouTube cross-check the IP mismatch. These paths now inherit the same
sessioned proxy as their player fetch.
Retry storms on dead IPs. The old two-level retry (tryMethod outside,
withRetry inside) burned up to 3 attempts on the same flagged session before
rotating. Collapsed to a single retry layer in tryMethod, which rotates
session on every retry — bot-block OR retryable network error.
Bot-block detection blind spots.isBotBlocked now matches HTTP 403/429/503,
empty caption bodies, consent walls, and "could not extract player response" —
the realistic Apify-residential blackhole signals, not just LOGIN_REQUIRED
text matches.
Insight 500 misclassification.classifyErrorStatus (the status reported to
Scraper Insight) didn't recognize the bot-block surfaces isBotBlocked already
catches — a raw HTTP 403/forbidden, "could not extract player response", and
empty caption bodies — so real bot-blocks were logged as generic 500s. They
now map to 403, and the 404 "unavailable" matcher was broadened ("video is
unavailable", "isn't available", "no longer available"). The 403 check runs
first so a bot-block's fake "video unavailable" still classifies as 403.
Added
ANDROID and IOS InnerTube clients, now leading the extraction chain. Measured
over 10 videos (caption actually fetched and parsed, not just a player OK):
ANDROID 10/10 with no visitorData, IOS 9/10, against ANDROID_VR — the previous
primary — at 1/10. Both return caption URLs without exp=xpe, so neither needs a
PO token. ANDROID needs no /embed preflight either, so most videos now resolve
on the first request instead of after a preflight plus 1-3 failed attempts.
Measured end-to-end on 8 videos (4 of which previously reached the yt-dlp
fallback): 26-28s with 0 preflights and 0 yt-dlp invocations, against 51-78s with
8 preflights and a yt-dlp fallback on the old chain.
Client versions refreshed from yt-dlp at startup.clientVersion staleness is
the biggest single cause of silent success-rate decay — the same ANDROID client on
the same video returns OK on 21.02.35, LOGIN_REQUIRED on 20.10.38, and HTTP 400 on
anything older, degrading on hard videos first so it rots invisibly. yt-dlp already
ships in the image and maintains this table, so refreshClientVersions() reads
INNERTUBE_CLIENTS at boot and re-points every client, rebuilding the User-Agent
in step (the native clients embed their version in the UA and YouTube cross-checks
the two). Falls back to the compiled-in literals. This immediately corrected three
hardcoded versions that were future-dated and invalid.
Under-collection warning for playlists. YouTube stops issuing continuation
tokens partway through large playlists (measured: uploads playlists stop at
~200 items). That server-side cap cannot be paged past, but the shortfall is no
longer silent — when a playlist advertises more videos than were collected, the
count of both is logged as a warning.
Pagination guards. Hard 50-page cap, stop-on-no-new-IDs, cross-page
deduplication, and a jittered 0.5-1s gap between continuation requests. A
mid-pagination failure now returns the pages already collected instead of
discarding them. paginate() takes injectable dependencies so the empty-page
recovery path is unit-tested deterministically — a live empty response requires
drawing a flagged proxy IP and cannot be forced on demand.
Proxy egress self-test on startup. Hits api.ipify.org through one
session and logs the exit IP, so the next time the proxy gets unplumbed it
shows up in the first log line instead of a multi-week Insight regression.
Cross-worker circuit breaker. When 5 consecutive videos fail across the
worker pool, all workers pause for 30s — prevents the thundering-herd burn
through proxy sessions when YouTube changes detection mid-run.
Optional apifyProxyCountry config. Pin Apify residential exit IPs to a
specific country (e.g. "US"). YouTube flags residential IPs from low-trust
regions ~5–10× more aggressively, so US/EU pinning typically pushes success
rate higher. Default is unspecified to avoid silently changing behavior for
existing users.
test-proxy-rotation.mjs diagnostic script. Probes the Apify proxy
directly: 5 fresh sessions should return 5 distinct exit IPs; the same
session ID called twice should return the same IP. Run via
npm run test:proxy inside apify run for credential injection.
PO-token provider — the real exp=xpe unlock. The image now bundles the
BgUtils PO-token provider (bgutil-ytdlp-pot-provider, script mode, pinned to
1.3.1 for both the yt-dlp Python plugin and the Node server build), and the
yt-dlp fallback routes minting through it via
--extractor-args youtubepot-bgutilscript:server_home=…. yt-dlp on its own
cannot mint the Proof-of-Origin tokens that gate exp=xpe caption URLs and the
"Sign in to confirm you're not a bot" wall — this is the only path that actually
unblocks those videos. The provider's BotGuard runner is built against
node-canvas, so the Alpine image carries the Cairo/Pango/JPEG/giflib stack at
runtime (toolchain + headers are build-only and removed). Server location is
overridable via the BGUTIL_SERVER_HOME env var.
PO-token provider self-test on startup. Runs generate_once.js --version
at init and logs bgutil <version> (or a clear warning), so a broken provider
surfaces in the first log lines instead of only when a hard video reaches the
fallback — mirrors the proxy egress self-test.
Changed
/embed preflight demoted to step 3 and its purpose corrected in the source. It
was documented as "the single highest-success-rate endpoint for transcript
scraping" with a caption fast-path, but returned zero caption tracks on every
video tested — the fast-path never fires. Its actual contribution is the
visitorData that lifts ANDROID_VR from 1/10 to 9/10, so it now runs only after
ANDROID and IOS have both failed.
Channel resolution now paginates the /videos tab instead of the uploads
playlist. The uploads playlist pages 100 videos per request vs ~30, but stops
issuing tokens at ~200 items; the /videos tab reaches the full list (verified
against yt-dlp: 448/448 and 801/801). The uploads playlist is kept as a fallback
for when the tab yields nothing.
InnerTube browse calls dropped the hardcoded ?key=AIzaSy… API key —
verified to return identical results without it, and one less credential to be
rotated out from under us.
got-scraping is now a declared dependency. playlist.ts always imported it,
but it resolved only transitively through apify and would have broken on any
dependency-tree change.
Per-request timeout bumped from 20 s to 35 s — residential-proxy realistic.
Inter-method delay bumped from 500 ms to 1500–2500 ms (jittered) and retry
cooldown from 500–1000 ms to 2000–4000 ms (jittered). Shorter cooldowns kept
pulling another IP from the same /24 that YouTube just flagged.
Debug logs now include [session=<id>] on every attempt and failure for
correlation against Apify's session-usage dashboard.
Cookie path pins one exit IP per video (sticky session). When cookies are
supplied, every method and retry for a video reuses a single proxy session
instead of rotating a fresh IP per attempt. Cookies carry a stable account
identity, and fanning it across many residential IPs in seconds is impossible
travel that flags the account. Without cookies, the rotate-per-attempt anonymous
strategy is unchanged.
