Proxy bypass via impit (critical).impit's per-request proxyUrl option
is silently dropped — it is only honored on the Impit constructor.
Previously every InnerTube and embed-page request was hitting YouTube directly
from the actor container's data-center IP, which is what tanked the 7-day
success rate to ~11.8%. Proxy is now passed at construction.
Session ID collisions under concurrency.newSession() used
Date.now() + 4 base36 chars (~1.6M values) — well into birthday-collision
range with the worker pool. Switched to crypto.randomBytes(8) (64 bits).
Player → caption IP split. Embed-FastPath and WEB-WatchPage previously
fetched the player JSON on session A and then the caption file on session B,
letting YouTube cross-check the IP mismatch. These paths now inherit the same
sessioned proxy as their player fetch.
Retry storms on dead IPs. The old two-level retry (tryMethod outside,
withRetry inside) burned up to 3 attempts on the same flagged session before
rotating. Collapsed to a single retry layer in tryMethod, which rotates
session on every retry — bot-block OR retryable network error.
Bot-block detection blind spots.isBotBlocked now matches HTTP 403/429/503,
empty caption bodies, consent walls, and "could not extract player response" —
the realistic Apify-residential blackhole signals, not just LOGIN_REQUIRED
text matches.
Insight 500 misclassification.classifyErrorStatus (the status reported to
Scraper Insight) didn't recognize the bot-block surfaces isBotBlocked already
catches — a raw HTTP 403/forbidden, "could not extract player response", and
empty caption bodies — so real bot-blocks were logged as generic 500s. They
now map to 403, and the 404 "unavailable" matcher was broadened ("video is
unavailable", "isn't available", "no longer available"). The 403 check runs
first so a bot-block's fake "video unavailable" still classifies as 403.
Added
Proxy egress self-test on startup. Hits api.ipify.org through one
session and logs the exit IP, so the next time the proxy gets unplumbed it
shows up in the first log line instead of a multi-week Insight regression.
Cross-worker circuit breaker. When 5 consecutive videos fail across the
worker pool, all workers pause for 30s — prevents the thundering-herd burn
through proxy sessions when YouTube changes detection mid-run.
Optional apifyProxyCountry config. Pin Apify residential exit IPs to a
specific country (e.g. "US"). YouTube flags residential IPs from low-trust
regions ~5–10× more aggressively, so US/EU pinning typically pushes success
rate higher. Default is unspecified to avoid silently changing behavior for
existing users.
test-proxy-rotation.mjs diagnostic script. Probes the Apify proxy
directly: 5 fresh sessions should return 5 distinct exit IPs; the same
session ID called twice should return the same IP. Run via
npm run test:proxy inside apify run for credential injection.
PO-token provider — the real exp=xpe unlock. The image now bundles the
BgUtils PO-token provider (bgutil-ytdlp-pot-provider, script mode, pinned to
1.3.1 for both the yt-dlp Python plugin and the Node server build), and the
yt-dlp fallback routes minting through it via
--extractor-args youtubepot-bgutilscript:server_home=…. yt-dlp on its own
cannot mint the Proof-of-Origin tokens that gate exp=xpe caption URLs and the
"Sign in to confirm you're not a bot" wall — this is the only path that actually
unblocks those videos. The provider's BotGuard runner is built against
node-canvas, so the Alpine image carries the Cairo/Pango/JPEG/giflib stack at
runtime (toolchain + headers are build-only and removed). Server location is
overridable via the BGUTIL_SERVER_HOME env var.
PO-token provider self-test on startup. Runs generate_once.js --version
at init and logs bgutil <version> (or a clear warning), so a broken provider
surfaces in the first log lines instead of only when a hard video reaches the
fallback — mirrors the proxy egress self-test.
Changed
Per-request timeout bumped from 20 s to 35 s — residential-proxy realistic.
Inter-method delay bumped from 500 ms to 1500–2500 ms (jittered) and retry
cooldown from 500–1000 ms to 2000–4000 ms (jittered). Shorter cooldowns kept
pulling another IP from the same /24 that YouTube just flagged.
Debug logs now include [session=<id>] on every attempt and failure for
correlation against Apify's session-usage dashboard.
Cookie path pins one exit IP per video (sticky session). When cookies are
supplied, every method and retry for a video reuses a single proxy session
instead of rotating a fresh IP per attempt. Cookies carry a stable account
identity, and fanning it across many residential IPs in seconds is impossible
travel that flags the account. Without cookies, the rotate-per-attempt anonymous
strategy is unchanged.
