# Changelog of Levels.fyi Scraper (`haketa/levels-fyi-scraper`) Actor

- **URL**: https://apify.com/haketa/levels-fyi-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/haketa/levels-fyi-scraper.md

## Changelog

### 1.1.0 — 2026-11-01

#### Fixed — major extraction rewrite (actor was returning empty/degenerate records)

- **Removed the dead `.md`-first strategy.** Levels.fyi's `/companies/{company}/salaries/{role}.md` endpoint now returns the **AWS WAF "Human Verification" challenge page**, not markdown. The old parser matched a stray `|` in that challenge HTML and saved a junk record (every compensation field `null`, `company` = the WAF page title) — and that false "success" blocked the working HTML path. The actor now loads the rendered salary page directly.
- **New primary parser: per-level roll-up from `pp.faqLevels`.** The Next.js `__NEXT_DATA__` no longer exposes individual submissions (`pp.averages[].samples[]` is empty, and samples are now redacted to total-comp only). The real per-level data lives in `faqLevels` — median total compensation, submission count and typical years-of-experience for **every level**. The actor now emits **one rich record per level** (L3–L9, E3–E9, SDE–Principal, …) instead of a single degenerate percentile row.
- **Enriched each level** by joining `pp.levels.levels[]` (titles, `percent_workforce`) and the page-level `pp.percentiles` (p10/p25/p50/p75/p90 for TC, base, stock, bonus).
- **Fixed the median-submission fallback** to read the current field names (`avgAnnualStockGrantValue`, `avgAnnualBonusValue`, `companyInfo.name`) instead of the obsolete `stockGrantValue`/`stock`/`bonus` keys.
- **Hardened WAF detection** in the page loader to also recognize the AWS WAF "Human Verification" challenge (previously only Cloudflare strings were detected).

#### Added

- New output fields: `levelTitle`, `typicalYoeMin`, `typicalYoeMax`, `percentWorkforce`, `sampleCountLast12Months`, `dataDate`. Updated `output_schema.json` and the dataset table view accordingly.

#### Verified

- Fresh verification run on 19 companies × 4 roles via residential proxy. Per-page extraction confirmed: e.g. Google · Software Engineer now returns **7 per-level records** (L3 $208.5K → L9 $2.36M median TC) with **100% coverage** on `company`, `level`, `levelTitle`, `totalCompensation`, `medianTC`, `p25TC`, `p75TC`, `levelSampleCount` and `percentWorkforce`; `typicalYoe*` present where Levels.fyi publishes a band. Zero `[object Object]` / scheme-less-URL / duplicate-key issues.

#### Docs

- Rewrote the README to the gold standard, grounded in the fresh run: accurate input/output tables, a real per-level output example, honest "present when available" notes, pagination-by-fan-out explanation, FAQ and related scrapers.
