# Changelog of YouTube Transcript Scraper & API - Transcripts + Comments (`superxcoder/youtube-transcripts-scraper`) Actor

- **URL**: https://apify.com/superxcoder/youtube-transcripts-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/superxcoder/youtube-transcripts-scraper.md

## Changelog

All notable changes to the YouTube Transcript Scraper & API Actor (`superxcoder/youtube-transcripts-scraper`, ID `zQzLQ5by5PAKCBrfj`).
Newest first. Dates are commit dates. Commit hashes refer to this repository. Platform build numbers are given where the docs record them.

### 0.2.5 (2026-10-24, commit pending)

#### Fixed

- **Comments were not billed with the live pricing.** Console saved the comment event as `Comment`; the Actor
  charged `comment`, which the platform treats as an unknown event (logged, nothing charged). Event names are now
  resolved against the configured pricing without regard to case or spaces (`resolveEventName` in
  `src/prices.ts`), for charging, budgeting and the setup check. Tests use the live configuration.
- Test P: 5 comments per video so the run stays under its $0.10 cap; pricing and charged counts compared without
  case.

#### Platform

- 2026-10-24: owner published the Actor with pricing `transcript` $0.002 and `Comment` $0.0005. `latest` moved from
  0.1.8 to 0.2.4 (`scripts/set-latest.mjs`), because Store users were getting 0.1.8.

### Earlier on 2026-10-24 (main)

- Launch price text: $2 per 1,000 transcripts (`transcript` $0.002), comments unchanged ($0.0005), in README,
  `.actor/actor.json`, `.actor/pay_per_event.json` and `PRICES_USD` (owner-approved 2026-10-24). No behaviour change:
  the Actor reads live prices from the platform.
- `scripts/cloud-release-test.mjs` test **P**, the priced test: refuses to start unless the platform pricing is
  exactly `transcript` 0.002 + `comment` 0.0005; then checks platform `chargedEventCounts` against the delivered
  rows, the $0.10 limit, the budget stop and duplicate handling.
- `scripts/build-info.mjs` (read-only: pricing, tags, build details) and `scripts/set-latest.mjs` (moves `latest`
  only with `CONFIRM=yes` and a SUCCEEDED build).

### 0.2.4 (2026-10-24, private build xJvh4EMhRwduLqqyj, commit 4f66338; release candidate)

#### Fixed

- Found by cloud test E on 0.2.3: a bot check inside the player's JSON answer (HTTP 200) was not counted as a block,
  so a run where every video was bot-checked would have failed with "the cause is the network or proxy". It is now
  counted (`transport.events.playerBotCheck`) and the run fails with "YouTube blocked every request (N bot checks,
  ...)".

#### Cloud tests

- D (15 geo/no-caption/ok videos, residential) and E (datacenter, bot-check path) on 0.2.3 and 0.2.4: 0 retries
  of any kind, bot-checked videos made one player request and got free `BLOCKED` rows, geo-restricted videos free
  `GEO_RESTRICTED` rows, billing tally = ok rows. Region coverage varies with the random proxy country (5 vs 9 of the
  same 15 geo-restricted).

#### Docs and tests (after the build, no behaviour change)

- STATUS.md rewritten for release: access policy, region policy, pricing review with margins at $1.50 and $2.00,
  Phase 5 validation table, Console steps. Billing tests for geo, blocked, PO-token and rate-limited rows (342
  tests). A code comment in `src/video.ts` updated.

### 0.2.3 (2026-10-24, private build 5VgXTF3XIFqJFpq3f, commit d430072)

Access policy: YouTube's refusals are accepted, not worked around.

#### Changed

- **Bot checks are final.** A bot check ("Sign in to confirm you're not a bot"), a captcha / sorry page, HTTP 403 or
  an HTML page instead of an API answer now raises `BotCheckError`: the video is not retried, the session (proxy IP)
  is not retired to get a new one, and no other player client is tried. The video gets a free `error` row with
  `errorCode: "BLOCKED"`. The same applies to listing and comment pages (a comments bot check keeps the transcript
  row and fills `commentsError`). Before: session retired, retried up to `maxRetries` times from new IPs, and the
  next player client was tried after a bot check.
- **PO-token captions are final.** `CaptionAccessError` (caption URLs answer with an empty body) is no longer retried
  on a new session; the row is `CAPTIONS_NEED_PO_TOKEN`. The empty caption body is now a `CaptionAccessError`
  inside the transport too.
- **No region hopping.** Geo-restricted videos were retried twice on new sessions to land in another country
  (`GEO_RETRIES`); they now get a `GEO_RESTRICTED` row at once. A user-chosen proxy country still applies.
- **Rate limits** (HTTP 429) keep the 1-30 s backoff and retries but no longer retire the session. Timeouts, 5xx
  and network errors are retried as before. Player client fallback stays for non-access failures (HTTP 400 for a
  retired client version, non-JSON, no playabilityStatus); when every client fails that way the error is plain
  (`HTTP_ERROR`/`UNKNOWN_ERROR`), not `BLOCKED`.
- Backoff for `blocked` is 0 (never retried).

#### Added

- Monetization guard (`pricingProblems` in `src/prices.ts`): a run fails before scraping anything when the Actor's
  pricing contains a priced `apify-default-dataset-item` event, because the platform would then bill every row,
  including free ones. A missing `transcript` or `comment` event is logged as an error.
- Regression tests for the access policy (bot check, 403, 429, PO token, geo, comments, fallback client) and the
  guard: 335 tests.
- `scripts/cloud-release-test.mjs` cases D (geo/no-caption videos of test C plus ok videos) and E (datacenter IPs,
  bot-check path).

#### Docs

- README, input and dataset schema text describe the new behaviour; Known limits has "Regions" and "Bot checks are
  not bypassed".

### 0.2.2 (2026-10-24, private build ucZYQlXK8zvKAeONI, commit 250f169)

#### Fixed

- The timeout stop message counted the run timeout from the Actor process start, so a 50 s timeout read "48 s"
  (cloud test A on 0.2.1). It now counts from `ACTOR_STARTED_AT`.

#### Added

- `scripts/cloud-release-test.mjs` (one pinned-build run at a time, verifies rows, segments, billing tally and
  cost) and `scripts/actor-info.mjs` (read-only Actor state).

#### Cloud tests

- A (old 50 s timeout input) SUCCEEDED with "Stopped before the 50 s run timeout" (0.2.1 and 0.2.2); B 20/20
  transcripts; C 91/100 (2 no captions, 7 geo-restricted, 0 failed). 184,371 segments validated. Details in
  `stress/results-cloud/2026-10-24/REPORT.md`.

### 0.2.1 (2026-10-24, private build Zor92P7bjdDZMlrd3, commit 4ddad4b)

Production pass. **Breaking output change** (the Actor is private and unpublished, so no users are affected): field
names now follow the spec.

#### Root cause of the ~50 s timeout

- The only run that timed out at ~50 s, `B4PIKXPBn4uJ01Fst` (build 0.1.8), was started with `timeoutSecs: 50` by
  the stress case `final-platform-timeout-hardkill` on purpose. It delivered 8 rows (all ok, 40 comments) and its
  RUN-SUMMARY matched the dataset.
- Real defects it exposed, fixed below: the run ended with Crawlee's "Crawled 9/44 pages" as the status message
  (nothing told the user what was delivered or why it stopped), and Crawlee's autoscaled pool starts at concurrency
  1 and ramps up slowly, so short runs used a fraction of `maxConcurrency` (9.4 videos/min in that run).

#### Changed (breaking)

- Row fields: `url` -> `videoUrl`, `channel` -> `channelName`, `error` -> `errorCode` + `errorMessage`,
  `transcript[{start,dur,text}]` -> `segments[{startSeconds,durationSeconds,text}]`, `isAutoGenerated` ->
  `transcriptType` (`manual` | `auto_generated`), `scrapedAt` -> `collectedAt`, chunk `start`/`end` ->
  `startSeconds`/`endSeconds`.
- New row fields: `transcriptAvailable`, `segmentCount`, `commentsError` (comment problems no longer go into the
  error field of an `ok` row).
- Input `maxVideos` is now `maxItems` (the old name is still read).

#### Added

- `errorCode` on every non-ok row: NO_CAPTIONS, AUTO_CAPTIONS_EXCLUDED, EMPTY_CAPTION_TRACK,
  LANGUAGE_NOT_AVAILABLE, VIDEO_UNAVAILABLE, LOGIN_REQUIRED, LIVE_STREAM, GEO_RESTRICTED, BLOCKED, RATE_LIMITED,
  CAPTIONS_NEED_PO_TOKEN, TIMEOUT, NETWORK_ERROR, HTTP_ERROR, UNKNOWN_ERROR.
- Inputs `includeTimestamps` (`[mm:ss] text` lines in transcriptText), `includeMetadata` (off skips the residential
  WEB player call and the date lookup), `videoTimeoutSecs` (per-attempt limit, default 120; each HTTP request gets
  at most a third of it, capped at 30 s), `maxRetries` (default 6).
- `src/reliability.ts`: exponential backoff with equal jitter by failure kind (bot check 0.25-1.5 s, 429 1-30 s,
  timeout/network/HTTP 0.5-8 s, PO token none), never sleeping past the timeout self-stop.
- Self-stop before the platform timeout (`ACTOR_TIMEOUT_AT` minus 5-15 s): no new deliveries, the one in flight
  finishes, STATE and RUN-SUMMARY (`status: "timed_out"`) are saved, the status message says how many videos were
  delivered and how many found videos were left (not charged). A timeout before any row fails the run.
- Progress status message and a 30 s progress log counting videos (not listing pages), with videos/min, retries and
  timeouts.
- RUN-SUMMARY `reliability`: retries by kind, timeouts, backoff time, per-video time (avg/p50/p95/max),
  videos/min.
- Transcript normalisation (`normalizeSegments`): times finite and never negative, invisible characters stripped,
  empty lines dropped, stable sort by start time.
- `scripts/bench-local.mjs` (13 local scenarios with live videos, peak RSS, stress checker);
  `scripts/list-runs.mjs`, `inspect-run.mjs`, `run-costs.mjs` (read-only platform run history, logs and costs).
- Tests: `test/round3.test.ts` (46 tests) and schema tests that require the dataset schema to list exactly the
  keys of a row. Vitest 313/313.
- README: error-code column, Known limits section, Python, RAG and n8n examples, a verified output row from a live
  run, retry/timeout notes, measured speed.

#### Fixed (found by the local regression suite, round 4)

- Saved inputs with the old `maxVideos` lost their limit: the platform fills in `maxItems`' schema default (1000),
  which won. The smaller of the two now applies.
- A "Maximum cost per run" could be exceeded by one comment event when the comment price was not loaded (local
  runs, where the SDK prices such events at $1): comment truncation treated the missing price as unlimited. Billing
  now budgets with the same effective price as the SDK.
- Stress case files and the checker used the 0.1 field names (`isAutoGenerated`, `transcript`, `channel`).

#### Changed

- Crawler starts at the full `maxConcurrency` (`autoscaledPoolOptions.desiredConcurrency`); it still scales down
  when overloaded. Local: 20 videos in 2.5 s at c10 (was ~5 s at every concurrency).
- STATE / RUN-SUMMARY are saved every 5 s instead of 2 s while rows are delivered (key-value writes were about
  $0.08-0.12 per 1,000 videos).
- Stress checker (`scripts/stress-run.mjs`) reads the new field names and also checks errorCode /
  transcriptAvailable consistency and segment order.
- `.actor/actor.json` version 0.2.

#### Docs

- `CLAUDE.md`: current state, how it works, source map, open items.
- `CHANGELOG.md`: this file, covering every commit since the scaffold.

### 2026-10-20: stress round 3 harness (fa1ccbd, c189226, c2615e8)

#### Added

- `scripts/stress-kill-resume.mjs`: versioned kill-and-resume billing check. Round 3: killed after 40 rows, resumed to 60 unique rows, 58 transcripts and 250 comments charged, no duplicates.
- `scripts/stress-edge.mjs` runs budget cases with local pay-per-event and the case's max total charge (`unitPrices`: $1 per event, else real prices).
- `scripts/stress-edge.mjs` takes `CASES_FILE` and `OUT`, so it can run the `cases-final-*` rounds.

#### Fixed

- `scripts/stress-run.mjs` `check()` assumed `outputFormat: "both"` when the input had none. The Actor's default is `text` since the UX pass, so 3 regression cases were wrongly flagged "transcript empty".

#### Results

- Local final edge round 3: 8/8 pass. Local regression round 3: 46/46.

### 2026-10-20: CLAUDE.md and lint (3d25a93, 6c224fe)

- Added `CLAUDE.md` with the Actor's purpose, demand background and folder rules.
- Lint: the WEB player starter in `src/video.ts` is async; ESLint ignores stress result files.

### 2026-10-20: hardening round 2, build 0.1.8 (45af6ff)

Code review round 2 (3 reviewers, 25 findings) and the one failed regression case (`platform-abort-graceful`).

#### Fixed

- **Caption access.** When a client lists no usable track, or its caption body is empty (PO token), every remaining player client is asked in order. Tracks that are PO-only on every client raise `CaptionAccessError` (one retry, then an `error` row) instead of a false `no_captions` row. `availableLanguages` keeps the first client's list.
- **Abort.** The `aborting` handler and the timeout timer close billing and wait for the delivery in flight before `STATE` and `RUN-SUMMARY` are written. Nothing is pushed or charged after the summary, and a resurrected run never bills a row twice.
- **Migration.** `addRequests` throws while migrating, so a listing page whose videos were not enqueued stays unhandled and runs again after resume; its claims and per-source counts are given back (`state.unclaim`).
- **Comments.** Every `next` request is bounded by the handler deadline. A failed continuation or "newest" sort keeps the comments already fetched and notes "comments stopped early" in `error`.
- **Run summary.** Charged counts persist in `STATE` (`state.addCharged`), so a migrated run reports billing for the whole run.
- **srv1 parser.** Decodes the XML level, strips styling tags, then decodes the HTML level; text like "x < 5" is no longer eaten.

#### Changed

- The residential WEB player call is skipped for caption-less videos that are dropped (`skipVideosWithoutCaptions`), saving residential traffic.
- Docs: CSV formula note, "channel not found" billing wording, pricing cost basis at the measured $9.3/GB, STATUS/PLAN at $1.50.

#### Tests

- `test/round2.test.ts`, `test/round2-state.test.ts`. Vitest 265/265.

### 2026-10-19: hardening round 1, build 0.1.7 (ec293cc, a23497d)

#### Changed

- **Pricing**: transcript $1.50/1K ($0.0015), comment $0.50/1K ($0.0005), no start fee. Replaces the $0.80/1K proposal, which lost money on long videos. New `src/prices.ts` reads live prices for the logs and `RUN-SUMMARY`.
- **Captions downloaded as srv1** instead of json3: identical segments, about 61% less residential traffic (automatic captions 2-3x smaller).

#### Added

- `src/lifecycle.ts`: migration and timeout handling.
- Final stress case sets: `stress/cases-final-edge.json`, `cases-final-load.json`, `cases-final-regression.json` (first committed as WIP in a23497d).
- Tests: `lifecycle`, `prices`, `pricing`, `schemas`; srv1 and json3 pair fixtures.

#### Fixed

- JSON/XML bodies are never treated as block pages.
- Comments with empty text are skipped.
- `languageMatch` is honest for the human-made fallback track.
- Schema and README wording; fixtures re-captured.

### 2026-10-19: UX pass (5c7e432, 9bb8ad9)

#### Changed

- Plain-language input schema titles and descriptions; default `outputFormat` is now `text`.
- README rewritten for the Store page; dataset schema output views (overview, transcripts, markdown, comments).
- Actor title/description: "YouTube Transcript Scraper & API - Transcripts + Comments".

### 2026-10-19: QA, build 0.1.3 (fd57429, ca92f95)

- Platform smoke run `xufGuEafacc6bvcqx` (1024 MB, 40 s) SUCCEEDED: 7 rows (5 ok, 1 no_captions, 1 unavailable), billing equal to the dataset, no comments billed on non-ok rows. Cost $0.0023.
- README: residential traffic per video corrected from "about 10 KB" to the measured 78-190 KB.
- `stress/results-*` added to `.prettierignore`; stress case files formatted.
- Added `STATUS.md` (verdict: ready as a private Actor).

### 2026-10-19: stress and review fixes, build 0.1.2 (614d0a9)

#### Fixed

- Channel handles that redirect to legacy vanity URLs resolve (one more `resolve_url` hop), e.g. `@LuisitoComunica`.
- Unknown channels/playlists fail loud; network/proxy failures get their own message and stop early.
- Hard per-request deadline for hanging or blackhole proxies.
- `deliver()` serialised so the budget is never exceeded; dedupe before delivery; comment paging deadline.
- Mobile client fallback on 4xx/5xx/non-JSON; client-aware caption tracks; PO-token detection.
- Comments fetched only on `ok` rows; fail loud when no video lists any caption track.
- Geo-block retry on a new session; WEB player retry and counter; BCP-47 language codes.
- Crawlee's terminal status message no longer overwrites the Actor's summary line.

#### Security

- Caption host allowlist; safe markdown headings.
- `scripts/sanitize-fixtures.mjs`: fixtures carry no commenter handles or visitor tokens.

#### Added

- `STRESS-REPORT.md`.

### 2026-10-19: stress harnesses (e5c6b14)

- `scripts/stress-functional.mjs`, `stress-edge.mjs`, `stress-load.mjs` with their case files and pre-fix reports under `stress/results-*`.

### 2026-10-19: first working Actor (23209ee)

Modules M1-M6 from `PLAN.md` integrated.

#### Added

- **Transport** (`src/innertube.ts`, `src/http.ts`): IOS/ANDROID/ANDROID_VR mobile players, WEB player for metadata, residential `video` pool and datacenter `discovery` pool, block detection, field masks.
- **Transcript** (`src/transcript.ts`): caption parsing, language selection with fallback (original / any / none), plain text and markdown chunks with deep links.
- **Parsers** (`src/parse.ts`, `src/listing.ts`): player, next, comments, channel tabs, playlists, search (old and new renderers).
- **Billing and state** (`src/billing.ts`, `src/state.ts`): pay-per-event, never bills non-ok rows, budget-aware comment truncation, persisted run state.
- **Orchestration** (`src/main.ts`, `src/routes.ts`, `src/video.ts`): `BasicCrawler` router, fail loud, `aborting` handler, `RUN-SUMMARY`.
- README Store page, dataset/output schemas, `scripts/stress-run.mjs` and `stress/cases.json`.

#### Fixed (integration)

- Skipped rows no longer count as pushed; router stats persist across migration; an unresolvable channel counts as a failed listing.

### 2026-10-19: scaffold (252ed6b)

- Probe results, plan and module specs in `PLAN.md`.
- Shared types (`src/types.ts`), input normalisation (`src/input.ts`), input schema, PPE pricing (no start fee), starter dataset and output schemas.
- Real Innertube fixtures (2026-10-19 captures), `scripts/probe-proxy.mjs`, `probe-web.mjs`, `capture-fixtures.mjs`.
- Project tooling: TypeScript, ESLint, Prettier, Vitest, Dockerfile, `AGENTS.md`.