# Changelog of Greenhouse, Lever, Ashby & Workable Jobs Scraper (Any Company) (`nekruz/ats-jobs-scraper`) Actor - **URL**: https://apify.com/nekruz/ats-jobs-scraper/changelog.md - **Full Actor documentation**: https://apify.com/nekruz/ats-jobs-scraper.md ## Changelog: ats-jobs-scraper ### 2026-10-13 - setup (Fable) - Initial version built and smoke-tested locally against live endpoints. ### 2026-10-13 - iteration 1 (loop) - **Cost/speed fix: Workable descriptions are now fetched lazily.** With `includeDescription: true`, the Workable fetcher used to request a detail page for every published job on the board *before* keyword, location, remote, date filters, dedup, `maxJobsPerCompany` and the pay-per-event charge limit were applied. A 200-job board with `maxJobsPerCompany: 5` made ~200 detail requests to push 5 jobs (and risked 429s). `fetchWorkable` now returns a `hydrate` step; `main()` calls it only for the jobs actually being pushed, after charging. Output fields are unchanged. - Test added: `workable descriptions fetched only for jobs that are pushed` (smoke.js). It uses the new `test/fetch-recorder.js` preload (`runActor(input, { recordRequests: true })` returns every fetched URL) and asserts 1 detail request for 1 capped job and 0 when the filter matches nothing. Verified it fails on the previous code (2 detail requests for 1 job) and passes now. `npm test`: 6/6, ~12 s. - Next candidates seen: Workable pagination has no 300 ms delay between pages and drops already-fetched pages if a later page fails; `fetchJson` ignores `Retry-After` on 429 and sleeps after the last attempt. ### 2026-10-13 - iteration 6 (loop) - **Workable pagination: early stop, 300 ms page delay, partial pages kept.** Workable's list endpoint returns 10 jobs per request. The fetcher paged the whole board back-to-back with no delay (a 300-job board \= 30 rapid POSTs, a 429 risk), kept paging after `maxJobsPerCompany` was already satisfied (live: `workable:upstream` with cap 1 made 2 list requests instead of 1), and if any later page failed it threw away every page already fetched and reported the whole board as `error`. Now: 300 ms between pages; pagination stops once `maxJobsPerCompany` new jobs pass the filters (same filter + dedup as the push step); a failure on page 2+ keeps earlier jobs and reports the board as `status: "partial"` with an `error` explaining which page failed (warning in the log). OUTPUT `jobsOnBoard` is now the board total Workable reports even when pagination stops early; new `jobsFetched` per board. Also `fetchJson` no longer sleeps after its final failed attempt. - Test added: `workable pagination: stops at the cap, pages fully otherwise, keeps pages when a later page fails`. Uses `workable:upstream` (12 jobs, 2 pages; must stay above 10 jobs): full run = ceil(total/10) list requests, cap 2 = 1 list request, and with page 2 forced to fail = status `partial` with the 10 page-1 jobs. Failure injection is new: `runActor(input, { failRequests: 'regex' })` makes `test/fetch-recorder.js` answer matching requests with a synthetic 503 (no network). Confirmed the test fails on the previous fetcher (2 list requests at cap). `npm test`: 7/7, ~19 s. - Next candidates: `fetchJson` ignores `Retry-After` on 429 (fixed 1 s/2 s backoff); a run that ends with 0 jobs pushed finishes with no warning; Workable detail hydration has no delay between detail requests. ### 2026-10-13 - iteration 11 (loop) - **Pay-per-event charge limit is checked before work and never exceeded.** The Actor fetched a whole board, then called `Actor.charge` for every job that passed the filters. When that count was above the user's remaining budget, SDK 3.7.2 charges one event past the limit (so the platform kills the run), and the Actor pushed that extra job. It also paged the full Workable board for jobs it could not charge, and with a budget below one job it still fetched the first board and charged $1 against a $0.5 limit. Now a `chargeableJobs()` check (the remaining 'job' budget) runs before each board (no request once the budget is spent), feeds Workable's early-stop `enough()` together with `maxJobsPerCompany`, and trims each board's jobs to the budget before charging. Descriptions are therefore fetched only for jobs that can be charged. Measured locally with a $3 budget on `workable:upstream` + `lever:palantir` with descriptions: before, 2 list + 4 detail requests and 4 jobs charged; after, 1 + 3 and 3, with Lever not requested. With $0.5: before, 3 requests and 1 job charged; after, 0 and 0. A bare slug skipped because the budget ran out is no longer reported as `not_found_on_any_platform`. - Test added: `charge limit caps list and description requests and never over-charges` (PPE simulated via `ACTOR_TEST_PAY_PER_EVENT` + `ACTOR_MAX_TOTAL_CHARGE_USD`; `runActor` gained an `env` option). Confirmed it fails on the previous code (4 jobs pushed against a $3 budget). `npm test`: 8/8, ~21 s. - Next candidates: `fetchJson` ignores `Retry-After` on 429 (fixed 1 s/2 s backoff); a run that ends with 0 jobs pushed finishes with no warning; Workable detail hydration has no delay between detail requests; a failed Workable description is still charged as a full job. ### 2026-10-13 - iteration 16 (loop) - **429/5xx retries honour Retry-After.** `fetchJson` retried a 429 after a fixed 1 s, then 2 s, whatever the server asked for, so a rate-limited board or Workable description request (Workable detail requests run back-to-back) usually burned all 3 attempts inside the limit window and was lost (board `error`, or a charged job without a description). It now uses `src/retry-after.js`, copied verbatim from seek-jobs-scraper: `Retry-After` as delay-seconds or any HTTP-date form, never shorter than the 1.5 s x attempt backoff, capped at 60 s. Retried response bodies are cancelled, 429 retries are logged as a warning with the requested wait, and there is still no sleep after the final attempt. 404 and other 4xx handling is unchanged. - Tests added: `Retry-After parsing: delay-seconds, all HTTP-date forms, bad values, cap` (unit) and `429 waits for Retry-After (seconds and HTTP-date) and never sleeps after the last attempt` (faked 429s on `workable:careers`: Retry-After 3 on the list request gives a ~3 s gap and the job; an HTTP-date ~8 s ahead on the description request delays the retry and the description arrives; persistent 429 gives 3 attempts with >= 4.5 s backoff, board `error: HTTP 429`, run ends right after). Confirmed failing on the previous `fetchJson` (1002 ms gap). `test/fetch-recorder.js` and `test/helpers.js` replaced by seek's extended versions (`failRequests {pattern,status,headers,times}`, `requestTimes`). `npm test`: 10/10, ~35 s. - Next candidates: a run that ends with 0 jobs pushed finishes with no warning (verified live: keyword that matches nothing on 2 boards -> "Done: 0 jobs from 2 boards", no warning); a failed Workable description is silently dropped and the job is still charged in full (no count in OUTPUT); no delay between Workable detail requests. ### 2026-10-13 - iteration 21 (loop) - **A failed Workable description is retried once later, and an incomplete job is no longer charged.** Verified live before the change: with every Workable description request failing (simulated pay-per-event), both jobs were charged, pushed with `descriptionPlain: null`, no error field, no warning, final status "Done: 2 jobs from 1 boards". Now `fetchJson` marks errors that survived all attempts on 429/5xx/network as `retryable`; those descriptions get one deferred retry after a 5 s pause at the end of the board (the pass stops at the first job that still fails); a 404 (job removed) or other 4xx is not retried. A job still without description is pushed with `descriptionError` and is not charged; descriptions are now fetched before `Actor.charge` (jobs were already trimmed to the budget, so no request is wasted). Workable description requests are spaced 300 ms apart like list pages (they ran back-to-back). OUTPUT gained `jobsCharged` (pay-per-event only), `descriptionFailures`, `descriptionsRecovered`, and per-board `descriptionFailures`; a warning and the final status message say "N without description (not charged), see descriptionError". Same pattern as seek iteration 18 / substack 19. - Test added: `failed workable description is retried once later, then delivered uncharged with descriptionError` (~30 s, ~3 live list requests): 503 x3 -> recovered after the >= 5 s pause, 4 detail requests, 1 charged; persistent 503 with cap 2 -> 2 items with `descriptionError: "HTTP 503"`, 9 detail requests, 0 charged, status text checked; 404 -> 1 request, 0 charged. Fails on the previous code (3 detail requests, no retry). `npm test`: 11/11, ~67 s. - Not changed: free incomplete jobs still count toward `maxJobsPerCompany`; README does not mention `descriptionError` or that such jobs are free (out of scope tonight). Remaining candidate: a run that ends with 0 jobs gives no clear status message or warning. ### 2026-10-13 - iteration 26 (loop) - **A run that saves no jobs now says why.** It used to end with only "Done: 0 jobs from N boards" and no warning, whether every job was filtered out, every board was missing, a board failed, or the charge limit was spent (verified live in iteration 16). Now the final status message and a warning list the reasons with counts, e.g. "Done: 0 jobs. 1033 jobs found but all filtered out (1033 by keywords) (see OUTPUT)" (live: `lever:palantir` + Stripe with a keyword that matches nothing). Reasons: charge limit reached before the first job; jobs found but filtered out, split by the first filter each job failed (keywords, locations, remoteOnly, postedAfter); boards with no open jobs; boards not found; company names not found on any platform; failed boards; unrecognized entries; partly fetched boards. OUTPUT gained `filteredOut` (jobs dropped per filter, for every run). Same pattern as app-store-reviews-scraper iteration 25. Runs that save jobs keep the old status message. - Test added: `a run that saves no jobs says why in the status message and a warning` (faked responses only, 0 live requests, < 1 s): 5 faked Lever postings each failing one filter -> exact message with per-filter counts and `OUTPUT.filteredOut`; 404 everywhere -> not found / not found on any platform / unrecognized; empty board; HTTP 400 -> failed board. The spent-budget case in the charge-limit test now checks its status message too. Fails on the previous code. `test/fetch-recorder.js` and `runActor` gained a `body` option for faked 200 responses (copied from app-store-reviews-scraper). `npm test`: 12/12, ~75 s. - Next candidates: with `includeDescription: false`, Greenhouse is still requested with `?content=true` (Stripe: 888 KB vs 34 KB compressed, ~1 s vs 0.45 s); dropping it loses `department` and `locations` (offices are only in the content response), so it would need the `/departments` and `/offices` endpoints (2 more small requests) to keep the output identical. Ashby has no parameter to omit descriptions (OpenAI ~13 MB), so nothing to save there. ### 2026-10-13 - iteration 31 (loop) - **Circuit breaker for Workable descriptions.** Iteration 21 made failed descriptions uncharged, but a dead detail endpoint was still hit 3 times (4.5 s of backoff) for every pushed job, plus one deferred retry: 5 jobs = 18 requests, 100 jobs = ~300 requests and ~8 minutes. Now, per board, after 3 descriptions in a row fail every attempt with 429/5xx/network, the rest are not requested and wait for the existing deferred retry (5 s pause). If that retry's first request succeeds, the skipped jobs are fetched too; if it fails, the board's descriptions are given up. Skipped jobs are saved uncharged with `descriptionError: "not requested: 3 descriptions in a row failed (HTTP 503)"`. 5 jobs with Workable down: 18 -> 12 detail requests; the saving grows with the board. 404s do not count toward the streak. OUTPUT boards gained `descriptionsStopped: true` when a board's descriptions were given up; the final status message then adds ", description requests stopped as Workable kept failing". Same pattern as seek iteration 23 / substack iteration 29 (per board, 3 in a row, one probe). - **Partly fetched boards show in the final status message.** A Workable board whose later list page failed (status `partial`, iteration 6) was invisible in "Done: N jobs from M boards" when jobs were saved. It now ends with "(N incomplete, see OUTPUT)", as in seek and substack. - Tests: the description-failure test now uses `workable:upstream` (needs >= 5 jobs): 9 injected failures with cap 4 -> breaker trips, job 4 skipped, deferred retry recovers all 4 (13 requests, 4 charged); permanent 503 with cap 5 -> 12 requests, jobs 4-5 "not requested", `descriptionsStopped`, status text checked. The pagination test checks "Done: 10 jobs from 1 boards (1 incomplete, see OUTPUT)". Both fail on the previous code (18 requests, no note). `npm test`: 12/12, ~84 s. - Checked, still open: the same board given twice (e.g. `workable:upstream` + its URL, cap 2) is fetched twice and, for Workable, the second pass pages on past the jobs already seen, so the run saves and charges 4 jobs for a cap of 2 (reproduced live: 3 list requests, 4 items); other platforms re-download the whole board (Ashby OpenAI ~13 MB) and add nothing. Fix: skip a repeated platform/slug with status `duplicate`. HTTP 200 with an HTML body is reported as board `error: non-JSON response` (verified, no crash) but is not retried. A whitespace-only entry is reported as `unrecognized`. A negative `maxJobsPerCompany` means no limit (schema minimum 0 blocks it on the platform). ### 2026-10-13 - iteration 36 (loop) - **Billing fix: a board entered twice was requested twice and, with a cap, charged twice.** Jobs were deduped, but every filtered job of the first pass was already marked seen, so the second pass over a Workable board paged on to new jobs: `workable:upstream` + its URL, cap 2 -> 3 list requests, 4 jobs saved and charged (reproduced live in iteration 31; reproduced again here with faked pages). Other platforms re-downloaded the whole board for nothing. Boards are now keyed by platform + slug; Greenhouse and Ashby ignore case (verified live: `STRIPE`, `OPENAI` return the boards), Lever and Workable do not (`PALANTIR`, `UPSTREAM` are 404), so only the first two are lowercased. A repeat (prefix vs URL, trailing slash, scheme-less URL, other case where the platform ignores it, the same bare name again) makes no request and is not charged; OUTPUT lists it as `{ status: 'duplicate', duplicateOf: '' }` with a warning. A bare company name is still probed once on all four platforms; once it resolved to e.g. Greenhouse, `greenhouse:` and its URL are duplicates of it, and repeating the bare name probes nothing. A bare name already not found anywhere is reported `duplicate` the second time instead of being probed again. Same pattern as ats-jobs-scraper-2 iteration 32 and substack-scraper iteration 34. - Careers URLs without `https://` (`jobs.lever.co/x`, `boards.greenhouse.io/x`, `x.workable.com`, `apply.workable.com/x`) were rejected as unrecognized or, for `x.workable.com`, probed as a bare slug on all four platforms; they are now parsed as URLs. - Workable: a job that shows up on two list pages (the list shifted between page requests) is kept once per board, so `jobsFetched` and the cap check no longer count it twice. - Test: the live `duplicate entries are deduplicated` test (which downloaded `workable:careers` twice) became `a board entered twice is requested and charged once; job ids deduped across pages`, faked responses only (catch-all rule, 0 live requests, < 1 s), simulated pay-per-event: Workable board in 4 spellings, cap 2 -> 1 list request, 2 charged (old code: 3 requests, 4 charged); uncapped with a job repeated on page 2 -> 2 list requests, 19 jobs, `jobsFetched` 19 (old: 20); bare name + prefix + upper-case scheme-less URL + Lever case variant + an unknown name twice -> 10 requests (old: 20) with the exact per-entry statuses. `test/fetch-recorder.js`/`helpers.js` ported from ats-jobs-scraper-2 (array of per-URL fake rules, JSON bodies). `npm test`: 12/12, ~88 s. - README does not mention status `duplicate` or `duplicateOf` (left for the docs pass). ### 2026-10-15 - pre-launch changes - **Retitled** to "Greenhouse, Lever, Ashby & Workable Jobs Scraper (Any Company)": the old title collided with existing Store listings (actor.json, README heading and the `publish.sh` Store copy updated). - Default run memory is now 256 MB (`defaultMemoryMbytes`); the maximum stays 1 GB. - The input form's `maxJobsPerCompany` is prefilled with 25, so a first "Try for free" run stays cheap and well inside the daily-test time limit. Leaving the field out through the API still means no limit (0). - Test fixture: `workable:upstream` now lists exactly 10 jobs (one page), so the pagination test moved to `workable:katmai` (63 jobs on 2026-10-15). The other tests still use `upstream`, which only needs 5+ jobs.