# Changelog of Substack Scraper – Posts, Comments, Publications & Authors (`teodor_banea/substack-scraper`) Actor

- **URL**: https://apify.com/teodor_banea/substack-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/teodor_banea/substack-scraper.md

## Changelog

### 0.1.5 (2026-10-08)

- The dataset batch window is 300 ms instead of 1.5 s. A handler waits for its batch
  while holding a concurrency slot, so a batch never held more than ~4 rows and the long
  window only slowed requests down (platform QA on 0.1.4: 2.9 s per request instead of
  1.1 s, $0.17 per 1,000 rows instead of $0.14).

### 0.1.4 (2026-10-08)

- Dataset rows are written in serialised batches (25 rows or 1.5 s; handlers wait for the
  write) with one pending-batch STATE write per batch. The previous per-push record was a
  single slot that concurrent handlers overwrote, so a resurrect could credit the wrong post
  and finish one row short; it also brought back one key-value write per row (+38% cost).
- A failed dataset write releases the batch's ids and claims so the retried handlers push
  the rows again instead of skipping them as "already pushed".

### 0.1.3 (2026-10-08)

- Restart reconcile decides from the dataset item listing at the saved offset instead of
  the dataset's `itemCount`, which lags real writes by several seconds on the platform
  (0 reported with 40 rows listed 1 s after a hard abort) and could have re-written a batch
  that was already there.

### 0.1.2 (2026-10-08)

- Each dataset write is recorded in STATE before it happens and reconciled against the
  dataset on restart: after a hard abort and resurrect the run's counters and status
  message now match the dataset (they were one row short).
- Publication rows from category leaderboards: `freeSubscriberCount` is documented as
  usually `null` — Substack only publishes the `rankingDetail` text there.
- Measured on build 0.1.3 (platform): $0.14 per 1,000 post rows with content, $0.044 per
  1,000 rows for posts with comments, no residential proxy traffic, $0.0005 floor.

### 0.1.1 (2026-10-08)

- Platform fix: through the Apify proxy tunnel Cloudflare answered HTTP 403 to impit's
  Chrome TLS profile on every `substack.com` host (publications and authors modes were
  unusable, subdomain inputs flaky); the Firefox profile passes, so the crawler now uses it.
- Resurrect/migration safety: claimed rows stay claimed (their detail requests are still
  queued), claims are persisted before the requests enter the queue, and pushed post ids
  are remembered so a request re-run after an interruption never writes a post twice.
- STATE and the status message are written at most every 5 seconds instead of after every
  row (per-row writes were 26% of a 500-post run's cost).
- Cloudflare blocks are reported as "blocked … retry later" rather than "not found";
  raw transport errors are truncated; postUrls mode reports failed URLs in the status.
- Hidden `{"debugProbe": true}` input runs a client × proxy matrix against Substack hosts
  and stores it under the `PROBE` key (diagnostics only).
- Measured on the platform: ~$0.20 per 1,000 post rows with content, ~$0.0015 per run
  floor; 500 posts with content in 6.5 minutes at 78 requests/min.

### 0.1.0 (2026-10-08)

- Initial release.
- Four modes over Substack's public JSON endpoints (no login): publication posts, specific
  post URLs, category leaderboards (publications) and author profiles.
- Post rows with full `bodyHtml` and readable `bodyText` (subscribe/share widgets removed),
  bylines, likes (heart reactions), comment and restack counts, tags, podcast fields,
  paywall flag and `contentStatus` (`full` / `preview` / `not_fetched` / `failed`).
- Optional comment rows flattened with `parentCommentId` and `depth`, capped per post.
- Custom domains resolved from the first redirect and cached in STATE; publication metadata
  read from the homepage's `window._preloads`.
- Verified endpoint behaviour recorded in `constants.ts`: archive `limit` max 50 but the first
  page returns 23 rows, adjacent pages may overlap by one row (offset advances by rows
  returned, ids de-duplicated, only an empty page ends the archive); `/api/v1/publication*`
  is 403; search endpoints are empty without a session (no search mode); a category page past
  the end answers HTTP 400.
- Per-publication and run-wide caps reserved synchronously; honest stop reasons
  (`exhausted`, `cutoff`, `pageLimit`, `failed`, `unreachable`, `unknownCategory`) in the final
  status message; failures under the `ERRORS` key-value-store key.
- Internally managed tiered proxying (datacenter → residential escalation) with impit Chrome
  TLS impersonation; no user proxy configuration.