# Changelog of Substack Scraper: Posts, Full Text, Comments & Podcasts (`nekruz/substack-scraper`) Actor - **URL**: https://apify.com/nekruz/substack-scraper/changelog.md - **Full Actor documentation**: https://apify.com/nekruz/substack-scraper.md ## Changelog: substack-scraper ### 2026-10-13 - setup (Fable) - Initial version built and smoke-tested locally against live endpoints. ### 2026-10-13 - iteration 4 (loop) - **Fixed: `search` silently returned only the first page.** With `search`, Substack's archive endpoint scans a fixed window of `limit` hits and returns only the posts that survive its own filtering (6, 9, 6, 7... of 20), in relevance order (it ignores `sort=new`). The Actor treated the first short page as end of archive, and with `postedAfter` stopped at the first older post. Lenny + "pricing" returned 3 posts (1 with postedAfter 2022) instead of 69. Search mode now keeps paging by window and stops after 2 consecutive empty windows; the short-page and postedAfter early stops apply only to the plain archive. - **Cost:** archive page size 12 -> 20 (Substack caps `limit` at 23). A full scan of a 482-post archive went from 41 to 25 requests; the type-filter test from ~31 s to ~9 s. - Checked: the archive endpoint has no working server-side type filter (`type=podcast` is accepted but ignored, other values are rejected), so `types` still filters locally. - Tests: new "search pages through short relevance windows" (15 results with search + postedAfter, and a no-hit search stops after exactly 2 requests; old code returned 1 result). The type-filter test now asserts page size 20 and floor(postsFound/20)+1 archive requests. Ported test/fetch-recorder.js + `recordRequests` in helpers. ### 2026-10-13 - iteration 9 (loop) - **Cost/reliability: the charge limit is now checked before work, not after.** With pay-per-event pricing, each archive page's posts are trimmed to what the user's max-charge budget still allows (`calculateMaxEventChargeCountWithinLimit('post')`) *before* bodies and comments are fetched and charged, and no archive page (or next publication) is requested once the budget is spent. Before, a whole page of details was fetched and then mostly thrown away, and `Actor.charge` was asked for more posts than the budget allowed, in which case the SDK deliberately charges one extra event so the platform kills the run. Local PPE simulation, $3 budget (3 posts), maxPosts 20, body + comments: 1 archive + 20 body + 20 comment requests, 4 posts charged and pushed (~30 s) -> 1 + 3 + 3 requests, 3 charged (~3 s). Budget below one post: 1 archive request + 1 post charged -> no requests, `chargeLimitReached: true`. Non-PPE runs are unchanged (budget treated as unlimited). - Test added: "charge limit caps body and comment requests and never over-charges" (ACTOR_TEST_PAY_PER_EVENT + ACTOR_MAX_TOTAL_CHARGE_USD; asserts 1 archive / 3 body / 3 comment requests, no request to the second publication, 3 full items, no over-limit charge attempt, and zero requests for a sub-one-post budget). The old code fetched 20 + 20 details and pushed 4. `runActor` gained an `env` option (ported from seek-jobs-scraper). - Observed: during this iteration's runs, a few body/comment requests were retried (transient 429/5xx) and one search test run returned 4 instead of 15 results (a failed later window ends the search). `getJson` still ignores Retry-After. ### 2026-10-13 - iteration 14 (loop) - **Fixed: a failed later archive page was reported as a complete run.** When page 2+ (plain archive or a search window) still failed after retries, the Actor kept the earlier posts but set the publication's status to `ok` with no warning, so a truncated run looked like a full one (seen live in iteration 9: a search returned 4 of 15 posts as `ok`). The publication is now `partial`, with `error` saying which offset failed and how many posts were kept (e.g. `archive page at offset 20 failed (HTTP 503); kept 20 posts from earlier pages`), a warning is logged, and the final status message says `(N incomplete, see OUTPUT)`. A 404 on a later page is treated the same way. Partial publications still count as scraped in the status message. A failed first page stays `error` / `not_found`. - **Speed:** `getJson` no longer sleeps after its final failed attempt (-4.5 s per permanently failed request). - Test added: "failed later archive page is reported as partial, failed first page as error". Injected 503s on Lenny's offset-20 page and every Stratechery archive request. It asserts 20 items kept, `partial` with the explanation, `error` for Stratechery, 3 attempts per failing page, the status message, and a run under 17 s. Old code reported `ok` and took 18.5 s. Ported the extended test/fetch-recorder.js and helpers.js (`failRequests`, `requestTimes`) from seek-jobs-scraper. - Not done: Retry-After on 429 is still ignored. Port ../seek-jobs-scraper/src/retry-after.js next. ### 2026-10-13 - iteration 19 (loop) - **Reliability/cost: a post whose body or comments could not be fetched is no longer charged as a complete post.** Before, a body/comment request that failed all 3 attempts left `bodyText`/`comments` null with no error field and no warning, and the post was charged in full (pay-per-event). Now `getJson` marks errors that survived every attempt on 429/5xx/network as `retryable`. Those details get one deferred retry at the end of the archive page, after a 5 s pause (`DETAIL_RETRY_COOLDOWN_MS`). The retry pass stops at the first request that still fails, so a Substack outage costs one extra request, not one per post. A 404, other 4xx or non-JSON answer is not retried. - A post still missing its body or comments is pushed with a new `bodyError` / `commentsError` field (e.g. `HTTP 503`) and is **not charged**. It still counts toward `maxPosts`. OUTPUT gained `postsCharged` (the SDK's charged `post` events, null outside pay-per-event) and per-publication `detailsRecovered`. `bodyFailures`/`commentFailures` now count final failures only. A warning is logged, and the final status message adds "N without body or comments (not charged), see bodyError/commentsError". Old code, all bodies 503: 2 of 2 posts charged with empty bodies. New: 0 charged, 9 body requests (2 x 3 + one 3-attempt deferred retry). First body 503 x3 then OK: body recovered (5 requests), 2 charged. - Test added: "failed body is retried once later, then delivered uncharged with bodyError" (PPE simulated; injected 503 x3 and persistent 503 on bodies, 404 on comments; asserts request counts, items, OUTPUT and status message). It fails on the old code. ~32 s, 3 live archive + 2 live body requests. npm test 8/8 in ~79 s. - Not done: Retry-After on 429 is still ignored (port ../seek-jobs-scraper/src/retry-after.js and the request loop from ../ats-jobs-scraper). The deferred retry's fixed 5 s pause may be shorter than a Substack rate-limit window. ### 2026-10-13 - iteration 24 (loop) - **Reliability: `getJson` now honours Retry-After on 429/5xx** via `src/retry-after.js` (copied verbatim from seek-jobs-scraper, same file as both ats Actors): delay-seconds or HTTP-date, never below the normal 1.5 s x attempt backoff, capped at 60 s. Before, a rate-limited archive, body or comment request was retried after a fixed 1.5 s and 3 s, so it usually failed again inside the limit window. Retried response bodies are cancelled; 429 retries are logged as warnings with the server-requested wait. Backoff without Retry-After is unchanged (1.5 s / 3 s), and there is still no sleep after the final attempt. - The deferred body/comment retry (iteration 19) now waits for the longest Retry-After seen on that page's exhausted requests when it is longer than the usual 5 s pause (capped at 60 s); `getJson` returns `retryAfterMs` with a retryable error for this. - Tests added: "Retry-After parsing: delay-seconds, all HTTP-date forms, bad values, cap" (unit, no network) and "429 waits for Retry-After on archive and body requests, including the deferred retry" (~28 s, 2 live archive + 1 live body request): faked 429 with an HTTP-date on the first archive request (old code waited 1.5 s, new ~5 s) and `Retry-After: 7` three times on a body (old gaps 1.5 / 3 / 5.25 s, new ~7 / 7 / 7.25 s, body recovered and charged). npm test 10/10 in ~90 s. - Trade-off: under a persistent rate limit with a long Retry-After, each failing detail request now waits up to 2 x 60 s (no circuit breaker yet). Seen in this iteration's baseline: "search pages through short relevance windows" once returned 10 of 15 results with no failed request recorded; it passed on 3 reruns (Substack search windows appear nondeterministic). ### 2026-10-13 - iteration 29 (loop) - **Test reliability: "search pages through short relevance windows" is now deterministic.** It ran against live Substack search, whose windows vary between runs; it failed once in iteration 24 (10 of 15 results) and would have cost the owner a false red `npm test`. All archive responses are now faked (0 live requests, ~3 s instead of ~4.5 s): short windows, an empty window between hits, a post older than postedAfter and a post repeated across windows. It asserts the exact posts (1,3,4,5,6) and offsets requested (0-100, ending after 2 empty windows), that maxPosts stops paging (offsets 0,20,40), and that a no-hit search makes exactly 2 requests. Checked live before the change: the Lenny "pricing" windows were identical across two scans and do repeat posts across windows (dedup needed). - `test/fetch-recorder.js` / `runActor`: `failRequests` now also takes a `body` (string or JSON) and an array of rules (first match with responses left wins), so tests can fake one archive page per offset. Env is one JSON `FETCH_FAKE_RULES`; existing string/object usages are unchanged. - **Reliability/cost: circuit breaker for body and comment requests** (ported from seek-jobs-scraper iteration 23), per publication and per kind. After 3 requests of one kind in a row failed every attempt with 429/5xx/network (`DETAIL_FAILURE_STREAK_LIMIT`), the rest of that kind on the page is not requested (posts get `bodyError`/`commentsError` "not requested: 3 body requests in a row failed (HTTP 503)" and go to the deferred retry). If the deferred retry's first request also fails, that kind is not requested again for the publication ("not requested: body requests kept failing (...)"); a success resets the streak and recovers the skipped posts. OUTPUT per publication gained `bodiesStopped` / `commentsStopped`; warnings are logged and the final status adds "requests stopped as Substack kept failing". Before: a dead endpoint cost 3 attempts (up to 2 x 60 s waits with a long Retry-After) for every remaining post. 22 posts with all bodies 503: 69 body requests -> 12, ~100 s -> ~24 s. - Test extended: the persistent-failure case of "failed body is retried once later, then delivered uncharged with bodyError" now fakes a 20 + 2 post archive and asserts 12 body requests, the per-post error texts, bodiesStopped, 0 charged and the status text (~24 s, was ~18 s for 2 posts). npm test 10/10 in ~109 s. - Still open: in search mode a window whose posts were all seen before resets the empty-window count, so if Substack ever ignored `offset` and kept returning the same non-empty window the run would loop until timeout (also a full non-search page of duplicates); a zero-result run ends without a reason in the status message; a non-array 200 archive response is treated as the end of the archive; free incomplete posts count toward maxPosts. ### 2026-10-13 - iteration 34 (loop) - **Billing fix: a publication entered twice was scraped and charged twice.** `seen` is per publication, so `fakepub`, `https://fakepub.substack.com/`, `FAKEPUB.substack.com` and `https://FakePub.substack.com/p/x` under simulated pay-per-event made 4 archive requests and pushed and charged 8 posts for 2 (verified before the change). Entries are now keyed by host (lower case; path, trailing slash and case ignored). A repeat is not requested and appears in OUTPUT as `{ status: 'duplicate', duplicateOf: '' }` with a warning. Redirects count too: when a subdomain's first archive request lands on a host already scraped (e.g. `lenny` -> `www.lennysnewsletter.com`), the entry is marked duplicate after that one request, nothing is pushed or charged, and later entries for either host are not requested. Live check: `lenny`, `https://www.lennysnewsletter.com/`, `lenny.substack.com` -> 1 request, 1 post charged. - **Loop guard:** an archive page holding only posts already seen (Substack ignoring `offset`) now ends a plain archive with a warning, and counts as an empty search window (2 in a row end a search). Old code paged until the run timed out. Also no archive request past offset 50,000 (`MAX_ARCHIVE_OFFSET`, warning). Status stays `ok` in both cases. - `test/fetch-recorder.js`: a fake rule's `url` sets the response's final URL, to simulate a followed redirect. - Tests added (both fully faked, no network, ~1.5 s together, both fail on the old code): "a publication entered twice is requested and charged once" (spellings + redirect-to-known-host case, request counts, postsCharged, OUTPUT statuses) and "an archive that ignores offset stops paging instead of looping" (plain: offsets 0,20; search: 0,20,40; old code hit the 20 s test timeout). npm test 12/12 in ~113 s. - Still open: a zero-result run gives no reason in the status message (no `filteredOut` counts yet); a non-array 200 archive response is treated as the end of the archive; free incomplete posts count toward maxPosts. README does not mention the `duplicate` status. ### 2026-10-15 - pre-launch changes - **Pay-per-event split:** runs with `includeBody: true` (the default) now bill the `post-with-body` event, and runs with `includeBody: false` bill the `post` event, so metadata-only runs no longer pay the full-text price. The charge budget check, the charge call and `postsCharged` in `OUTPUT` all use the same event. README Pricing updated. - **Retitled** to "Substack Scraper: Posts, Full Text, Comments & Podcasts": the old title collided with an existing Store listing (actor.json, README heading and the `publish.sh` Store copy updated). - Default run memory is now 256 MB (`defaultMemoryMbytes`); the maximum stays 1 GB.