# Changelog of Jobs Feed API - Deduplicated LinkedIn, Indeed & ATS Jobs (`sergeyfaraday/jobs-feed-api`) Actor

- **URL**: https://apify.com/sergeyfaraday/jobs-feed-api/changelog.md
- **Full Actor documentation**: https://apify.com/sergeyfaraday/jobs-feed-api.md

## Changelog

### 0.7.0 - 2026-08-21

- **Browser transport for Indeed and Glassdoor.** Both guest sources now read through a Camoufox browser instead of the plain HTTP client: the 0.6.0/0.6.1 smokes found Cloudflare serving a managed challenge on the HTTP path - for Glassdoor on every reachable platform exit (built-in residential in any country, the auto group, the container egress), for Indeed on the built-in residential exits tested - while the follow-up Camoufox spike passed both sites clean. The browser is launched lazily - only when Indeed or Glassdoor is selected - so ATS-only runs (Greenhouse/Lever/Ashby) are untouched: no browser, no extra memory, no extra startup time. Cold start on a fresh worker adds about 20 seconds; the memory gate `GUEST_NEEDS_MEMORY` stops a run before scraping if it has less than 2,048 MB and Indeed or Glassdoor is in scope. Rollback env `JOBS_FEED_GUEST_TRANSPORT=http` restores the old HTTP path per-run if the browser path regresses. Live Indeed smoke through the browser transport: 45 listed, 14 matched, no challenge, 51 s run, 803 MB peak.
- **Glassdoor is back in the listing** (schema enum, store title/description, README) now that the browser transport clears its Cloudflare challenge. Behaviour unchanged from 0.6.0: first search page per query and location inside your `datePosted` window, up to 30 jobs, snippet descriptions, contributes no removal signal.
- Indeed still reads one guest page per query (page 2+ is a login wall on Indeed - see 0.6.1); unchanged by the transport switch.
- LinkedIn (public) is unchanged: it still reads through the plain HTTP guest API, not the browser transport.
- `costTelemetry.perSource` now carries browser request/byte accounting (requests, HTML characters, wire bytes, challenges, failures) alongside the existing HTTP accounting, for Indeed and Glassdoor.

### 0.6.1 - 2026-08-20

- **Pipeline ~12x faster on large runs** (5K postings 64.6 s -> 7 s, 10K 12 s): per-shingle sha1 memoised in `simhash64` (bit-identical output, reference test), shingle-set hash switched to FNV-1a (membership-only, containment verified numerically equivalent on the calibration corpus). Dedup thresholds untouched.
- Proof tests for the README claims (CI-blocking): four-run duplicate simulation, three-source triple = one charge, no per-run minimum, proxy selection (extracted to `src/http/proxySelect.js`, behaviour unchanged), 10K-row memory ceiling, CSV/XLSX export smoke tool (`tools/deploy/csv-export-smoke.js`).
- Listing: README restructured (sources, delta, no-login, n8n, MCP, legality per source, current limits, pricing); store title "Jobs Feed API - Unique LinkedIn, Indeed & ATS Jobs, Deduplicated (No Login)". Glassdoor hidden from the schema, title and README while paused (see 0.6.0).
- 30-day cost table states its multiples against the sampled listing prices (6x at the median), not an unsourced range.
- Indeed reads ONE guest page per query: page 2+ (`start=10`) is a login wall on Indeed (Camoufox spike 2026-08-20), so it is never requested (D18). Was "up to 10 pages" in the 0.6.0 note. Store title shortened to the platform's 63-character cap.

### 0.6.0 - 2026-08-20

Sources research + a compliance fix found by it (`docs/specs/2026-08-20-sources-glassdoor-trueup-design.md`).

- Perf: pipeline throughput ~12x on large runs (5K postings 64.6 s -> 5.2 s) - per-shingle sha1 memoised in `simhash64` (bit-identical output) and the shingle-set hash switched to FNV-1a (membership-only; containment verified numerically equivalent on the calibration corpus). Dedup thresholds untouched.

- **Glassdoor: attempted, paused, hidden.** (Owner call 2026-08-20, after the smokes below.) The source is removed from the input schema enum, the store title/description and the README until a working access path exists; the adapter, its tests and the operator kill switch stay in the build so nothing has to be rebuilt when one does. Original note: The adapter is complete and tested, but the 0.6.0 smokes showed Glassdoor serving a Cloudflare managed challenge to every exit reachable from the platform (built-in residential in any country, the auto group, the container egress) on the first request, while a consumer IP is served normally. The operator kill switch `JOBS_FEED_DISABLE_GLASSDOOR` is set in Console, so selecting `glassdoor` yields `sourceHealth.status: disabled` and costs nothing. It is re-enabled when a working path exists (own proxy via `proxyConfiguration.proxyUrls`, or a browser fallback after its own cost/class decision). Details: `docs/deploy-checklist.md`.

- **Indeed search is now robots-compliant (behaviour change).** Our search URLs put `start` last, so Indeed's `Allow: /*&start=0&` … `/*&start=90&` never matched and the blanket `Disallow: /*&start=` applied - every search request, page 1 included, was disallowed. `searchUrl()` now emits `?q=…&start=N&sort=date` (+ `l=`/`fromage=`) and pagination stops at the real allowance of 10 pages, down from 30. **Consequence for users:** up to 100 listings per search term instead of 300, ordered by date rather than relevance. For a delta feed (what changed since the last run) this is close to neutral; for a first backfill it is a genuine reduction. `/viewjob` is disallowed and is emitted as a link only - we never fetch it.

- **Robots watch in the daily canary.** `tools/robots/check.js` re-asserts, for `glassdoor.com` and `indeed.com`, every access fact our adapters were designed against; it runs in the `source-health` workflow and fails the run when a rule we depend on moves. Snapshots in `docs/robots/`; `tests/unit/robots.adapters.test.js` checks every URL the adapters can construct against them, so CI fails too. Assertion-based, so cosmetic edits stay quiet.

- Research outcome: **TrueUp / Lenny's Jobs is not built** - the only lawful access door is their $1,000/month data licence; there is no unsigned public job surface, and both robots.txt and the Terms exclude automated agents by name.

- **Glassdoor (opt-in guest source, spec 2026-08-20 S1).** First search page per query × location × `datePosted` window (30-day ceiling when unset), up to 30 jobs each; never paginates and never opens detail pages (both robots-disallowed - asserted in tests against the stored snapshot); snippet descriptions; p10-p90 salary with `estimated`; `easyApplyFlag`. Contributes no removal evidence. Kill switch `JOBS_FEED_DISABLE_GLASSDOOR`. Locations resolve through a small hand-maintained token table (`data/glassdoor-locations.json`); unknown locations fall back to a United States search filtered by the card location (`GLASSDOOR_LOCATION_UNMAPPED`). New codes `GLASSDOOR_CHALLENGED`, `GLASSDOOR_SLICE_TRUNCATED`, `GLASSDOOR_LOCATION_UNMAPPED`.

- Sources enum order now encodes the basis hierarchy: ATS trio, then Indeed → Glassdoor → LinkedIn.

### 0.5.1 - 2026-08-19

Feature-gap P0 (frozen spec §3, owner review): **location + datePosted** - table stakes a jobs product cannot ship without.

- `location` (top-level) and `datePosted` (`any`/`24h`/`3d`/`7d`/`14d`/`30d` or integer days 1-365) scope the whole run; per-query objects `{"q", "location", "datePosted"}` override them (spec §3 object queries; console keeps flat fields, the API accepts both).
- Guest sources push the filters into the search URL (LinkedIn `location=` + `f_TPR=r{s}`, Indeed `l=` + `fromage=`) and trust the platform's geo resolution; ATS boards filter client-side (location substring over the posting's location text; posted/updated date cutoff, dateless postings kept). `datePosted` doubles as the cost control - no more paying to re-scan history.
- Removals stay honest under filters: an ATS posting seen by the scan but dropped by the scope is recorded observed-alive (never a removal candidate); guest scans that carry a location/date window are never conclusive (a job aging or moving out of a window is not a disappearance).
- `scopeFingerprint`: filters enter the canonical scope only when set - existing delta states (including the soak store) keep their fingerprint byte-for-byte; scan keys gain `|loc:`/`|age:` suffixes only for filtered searches.
- Input schema: `location` + `datePosted` in "What to search"; `proxyConfiguration` prefill dropped (the built-in residential default needs no toggle); first-click math updated to the new price grid ($0.28 at 200 rows).
- Golden: +6 cases (filters group) - parsing/overrides, scope semantics, guest URLs, fingerprint stability, removals × datePosted end-to-end.
- **Pricing update (§1 grid, registered on the platform 2026-08-19, pricingInfos record 4):** unique-job $0.0014/0.0013/0.0011/0.0010, job-update $0.0012/0.0011/0.0010/0.0009 (a change signal, ~0.86 of unique - not a cheaper row), job-removed $0.0020/0.0018/0.0016/0.0015 (premium closure signal); job-detail-fetch registered dormant at the A3 placeholder. Grid source of truth: `docs/specs/2026-08-18-pricing-update-spec-s1-grid.md`; `tests/ci/pricing-parity.js` fails CI on any divergence across pay\_per\_event.json / events.js / README / input schema. Money table and first-click math updated ($0.70 day 1, ≈ $1.7/month scenario, $0.28 first click).

### 0.5.0 - 2026-08-19

Spec amendments batch 2026-08-18 (`docs/specs/2026-08-18-spec-amendments-batch.md`): Part A proxy & detail economics + Part B WS6 jobseeker axis.

- **A1 - bundled proxy by source class** (amends D13): LinkedIn/Indeed default to the built-in Apify residential proxy; ATS boards stay direct. `proxyConfiguration` moved to Advanced as the power-user override ("bring your own proxy") - never removed, it is the margin escape hatch.
- **A2 - LinkedIn default = list mode**: full posting pages are never fetched implicitly. New `detailFetch` input: `off` (default) / `all` / `selective` (+ `detailFetchMinScore`); the spec object form `{ minProvisionalScore }` is accepted in the API.
- **A3 - `job-detail-fetch` billable event** (PLACEHOLDER $0.0020 FREE → $0.0015 GOLD\_PLUS): one per full posting page fetched from a guest source, charged once per run (`{runId}:job-detail-fetch`), never in the default list mode. Price finalized only after costTelemetry measures real detail-page weight through residential (decision-18 margin gate applies independently). Register on the platform BEFORE pushing this build. A charge failure never kills a delivered run: `DETAIL_FETCH_CHARGE_FAILED` warning, our loss. Report: `costTelemetry.detailFetchesCharged`.
- **B1 - `matchProfile`**: whole-word, case-insensitive keyword scoring with aliases and weights (`matchScore = matched weights / total weights × 100`, integer); phrase semantics for multi-word keywords. Row fields `matchScore`, `matchedKeywords`, `unmatchedKeywords`, `matchBasis` (`title+snippet` list mode vs `title+description` - the honesty marker). Rows below `minMatchScore` are **not emitted and not billed** (report `matchFiltered`); `minMatchScore: null` annotates only.
- **B2 - match filtering × delta state** (amends D10 state shape): filtered rows enter state as `emitted: false` (insert-if-absent, no snap); a known-but-never-emitted occurrence that later passes emits as `unique-job` with the same occurrence id and preserved `firstSeenAt`; a filtered observation of an emitted occurrence suppresses the row without rebaselining (the pending change still emits when it passes). Removal tracking and snaps run only for `emitted: true` occurrences.
- **B3 - selective detail fetch**: two-pass flow - provisional score on title (list mode) gates the full-page fetch (`detailFetchMinScore`); `minMatchScore` applies to the FINAL score on full text. A fetched page is billed even when the final score then filters the row - you paid to find out (documented in README).
- **B4 - signal fields**: `applicantsHint` (strictly from "Be among the first N applicants" on LinkedIn detail pages), `easyApplyFlag` (Indeed), `earlyApplicantsOnly` input filter (filter-before-billing, same as B1).
- **B5 - jobseeker recipe**: n8n template `templates/n8n/jobseeker-digest.json` (schedule → delta run with matchProfile → route matchScore ≥ 80 to a digest); README "For job seekers" section with the $2-4/mo math.
- Golden: +9 cases (score math, filter-before-billing, emitted:false lifecycle, matchBasis, selective fetch billing, removal skip, earlyApplicantsOnly); CI event-set assert now 3 row events + 1 service event; `EVENT_ORDER` stays row-only.

### 0.4.3 - 2026-08-19

- Input schema hygiene for the Actor quality checker (it flags field names/texts containing Key/Token as secrets): `companies[].greenhouseBoard` is the console name for the Greenhouse board slug (legacy `greenhouseToken` still accepted in the API); descriptions no longer say key-value / keywords / tokens. No behaviour change.

### 0.4.2 - 2026-08-19

- Catalog 2026.08.3 from the new `ats-catalog-builder` actor (plan Phase F; public discovery from the yc-oss companies list + explicit seeds, verified against the board APIs): **704 boards** (197 Greenhouse, 427 Ashby, 80 Lever; 23.3K postings), 188 with a verified board ↔ company domain link; unverified boards carry no domain and are named from the board itself (label collisions like a YC "Pulse" vs "Pulse Healthcare" stay apart). Catalog schema: `domain` nullable, `domainVerified`; the resolver answers only from verified entries; ATS adapters take the catalog company name when the board has none (Lever/Ashby). `tools/catalog/pull.js` syncs `catalog-latest` into the build; builder scheduled weekly (Sunday 03:00 UTC).

### 0.4.1 - 2026-08-18

- WS5-A live source health (plan Phase E): `tools/health/publish.js` aggregates the owner canary tasks (`source-health-canary` 2×/day GH + LinkedIn, `removals-soak-canary` daily) into `docs/health/source-health.{md,json}` and a public gist (anonymous read verified); README links it; `.github/workflows/source-health.yml` refreshes daily once `APIFY_TOKEN` / `GIST_TOKEN` secrets are set.

### 0.4.0 - 2026-08-18

- `seedFromDataset` migration kit (УТП spec WS4, plan Phase D; spec §3/D10 amendments in §14): resourcePicker(dataset, READ) + `seedFormat` auto / jobsFeed / linkedinScraperCompat / indeedScraperCompat; seeds an EMPTY delta state for free (revision 1, no rows, no events) so the first live run emits only genuinely new jobs. Preconditions `SEED_NEEDS_DELTA`, `SEED_STATE_NOT_EMPTY`, `SEED_TOO_LARGE` (50,000), `SEED_FORMAT_MISMATCH` (≥ 5 % unparseable) - all pre-scrape, $0. Seeded occurrences: `seeded: true`, `firstSeenAt` = their postedAt, `lastSeenAt` = seed time (T18), aliases as a live observation would derive them, null hashes for absent fields (baselined silently on first sight - null-hash rule in `changedFields`), `seenVia` empty (never removable until observed live), snap from the seed. Report `seed`, `deltaStats.seeded / seedSkipped`.
- Fix: unresolved company domains (no ATS board found) stayed out of the run entirely, so a LinkedIn-only run for such a company lost its company filter; they now stay in scope without board tokens (ATS adapters skip them, guest sources filter by them).

### 0.3.1 - 2026-08-18

- WS3 schedule-first onboarding (plan Phase C): README leads with the 30-day cost table (footnoted with Store prices of the day and a real delta pair) and the 3-minute daily-feed walkthrough; one-shot demoted to "Try it once". n8n template `templates/n8n/daily-jobs-feed.json` (schedule → run task → route new / updated / removed). Report `meta.origin` (WEB / API / SCHEDULER) and OUTPUT `origin`; OUTPUT/log `nextStep` nudge on non-scheduled or non-delta runs.

### 0.3.0 - 2026-08-18

- `job-removed` (УТП spec WS1, plan Phase B; spec amendments D1/D9/D10 in §14): opt-in `trackRemovals` + `removalConfirmRuns` (delta mode); conclusive-scan rule over per-occurrence `seenVia` scan keys (≤ 8 per source) and per-run scan completeness from every adapter; `removalConfidence` high (ATS boards) / best-effort (LinkedIn/Indeed) with a first-run `REMOVALS_BEST_EFFORT_SOURCE` warning; removal rows carry the last known summary from separate snap shards (own 60 MB budget, LRU, fallback row when evicted; `snapshotAvailable`), `removedAt`, `daysOpen`; `deltaStatus: removed` in both dataset views; emission priority new > updated > removed; a removed identity seen again reopens as a new occurrence with `repostOf` (billed unique-job).
- New billable event `job-removed` ($0.0004 FREE / 0.00035 / 0.0003 / 0.00025 GOLD+), registered with this build; CI asserts the three-event set.
- Delta state: bytes-driven eviction now converges in one persist (previously 10 % of occurrences per persist regardless of size). Measured: a real occurrence ≈ 1.26 KB, so the 50 MB core holds ≈ 37K occurrences (finding F6); `LIMITS.snapMaxBytes` 60 MB for removal summaries.
- Report: `deltaStats.removed / removalsExamined / removalsInconclusive / reopened`, `sourceHealth[].scans`.

### 0.2.0 - 2026-08-18

- Catalog mode (УТП spec WS2, plan Phase A): with no `companies`, ATS sources enumerate the bundled catalog of verified boards (`data/ats-catalog.json`, `catalogVersion` 2026.08.1, 44 boards) largest-first within `maxScannedJobs`; new input `catalogMode` auto/off; report field `catalog`; optional operator KVS live override (`CATALOG_KVS_ID`, warning `CATALOG_OVERRIDE_UNAVAILABLE`), `CATALOG_UNAVAILABLE` only for a corrupt bundle. The resolver answers from the catalog before probing (`companies.resolvedFromCatalog`).
- Query filter for ATS boards matches the title (all tokens) OR the description as a phrase (tokens adjacent, in order - scattered tokens matched too loosely on the first platform smoke: recruiters and design engineers for "senior backend engineer"); under `maxItems` title matches are emitted before description matches (deterministic). Guest sources stay title-only at list time.
- Console prefill `maxItems: 200` (first click ≈ $0.18); API default unchanged (1000).
- `tools/catalog/build.js` + `seed-boards.txt`: verify boards live and write the catalog (Phase F replaces the seed with public discovery).

### 0.1.6 - 2026-08-18

- Store discount tiers registered (Bronze/Silver/Gold+; Free unchanged). Example task prepared for publishing.

- Dedup: aggregator fan-out is same-country only (a LinkedIn card in another country with a region suffix in the title stays a separate, annotated row). Cross-source holdout (141 pairs, human-confirmed + rule A): pairwise precision 0.957 (4 "fp" = D7-resolved twin cases), recall 1.0.

### 0.1.5 - 2026-08-18

- Dedup: aggregator fan-out (spec §14) - LinkedIn/Indeed cards of one posting in several cities join the ATS group (D7 exempt for aggregators; location veto becomes a ranking penalty for aggregator pairs). `location.all` = union of member cities; per-source LinkedIn hashes cover the city set. Dense smoke: 357 of 403 unmerged LinkedIn rows were this class.

### 0.1.4 - 2026-08-18

- Location parser: "In-Office" / "On-Site" / "In-Person" are work modes (were parsed as IN/ON + city after the prefix rule).

### 0.1.3 - 2026-08-18

- Location parser: `US-`/`CA-` prefixes, city shorthand (SF, NYC, SEA, CHI, ATL, DC…), comma lists of places as multi-locations, LinkedIn "New York, United States" as the city. Blocking adds a company|title-token block so cross-source listings with unparsed locations still meet (dense smoke: 293 Stripe pairs never met).
- Cross-source labeling tool `tools/corpus/xsource-pairs.js` (stratified sample) and `tests/corpus/labels-xsource.csv` (150 pairs, awaiting labels).

### 0.1.2 - 2026-08-18

- Commit chain: dataset offsets no longer trust the platform itemCount alone (eventually consistent - returned 0 after 500 verified rows and stopped batch 2 with DATASET\_COMMIT\_AMBIGUOUS); the journal/in-process floor wins.
- Dedup: strong/probable descriptions also match by shingle containment (shorter text ≥ 90% contained in the longer, ≥ 60 shingles) - LinkedIn guest pages append company boilerplate to the ATS text, which drifted the simhash 11-28 bits at identical titles/cities. Location veto and Hamming thresholds unchanged; dev fp still 0/221.
- README: maxDetailFetches note (LinkedIn cards without a fetched detail cannot merge with their ATS twin).

### 0.1.1 - 2026-08-18

- Dedup: location veto in the strong tier - identical title+description in different cities/countries is annotated as probable, never merged (`locationConflict`). Holdout confirmed by Sergey: fp=0/57, tp=2/2 (docs/dedup-calibration.md).
- Normalizer: with a concrete city in the location, only `#LI-Remote` / `#LI-Hybrid` in the description may override the work mode; phrase heuristics apply only when no city is named (a "four weeks of fully remote work" perk no longer flips a hybrid role to remote).

### 0.1.0 - 2026-08-17 (platform builds 0.1.1-0.1.5, actor `sergeyfaraday/jobs-feed-api`)

- ATS sources: Greenhouse, Lever, Ashby (public board APIs) + resolver from company domain.
- Normalizer built on a 6.5K-posting corpus: salary from pay-transparency text, multi-location parsing, work mode, geoBucket, company identity (D5).
- Dedup with exact/strong/probable tiers and the D7 component invariant; calibration corpus (dev/holdout).
- Delta mode: sharded, revisioned state with a single meta pointer commit; sentinel head-lock queue; alias index; coalescing; 45-day reposts; lazy re-baseline; generations with rebase.
- Commit chain: immutable batch payload + manifest journal, prefix-verified push, state before charge, idempotent charge keys, same-run resume; mutually exclusive `unique-job` / `job-update`.
- SSRF-hardened HTTP client (`publicFetch`) for every user-influenced URL.
- PPE registered 2026-08-17 (unique-job $0.0009, job-update $0.0005).
- LinkedIn public (guest) source; Indeed opt-in (residential proxy required, experimental); `proxyConfiguration` input for the guest sources.

### Unreleased

- 2026-08-17: repo bootstrap from spec v1.3 - AGENTS.md, Apify platform playbook, .actor templates, module skeleton, codes, CI (synthetic-events assertion).