# Changelog of Idealista Property Scraper (`devilscrapes/idealista-property-scraper`) Actor

- **URL**: https://apify.com/devilscrapes/idealista-property-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/devilscrapes/idealista-property-scraper.md

## Idealista Property Scraper — Changelog

### 0.2.0 — 2026-09-22

- **Fixed the real defect behind the fleet audit's "17 fields dead across every
  row" finding (build 0.1.3, dataset `YopOOwkIgdwAJSZBf`) — never-implemented,
  not a regression.** A live Camoufox recon (search + detail page, ES/IT/PT,
  same warm-up/proxy path as production) found idealista serves **zero**
  `application/ld+json` and **zero** `#agency-data` scripts anywhere — the
  entire previous parser (`src/parser.py`) targeted a wire format the site
  never actually served, built against a hand-authored fixture that invented
  it (`reference-hand-authored-fixtures-encode-fiction`). Full writeup:
  `docs/specs/idealista-property-scraper/notes.md`.
- **`src/parser.py` rewritten against the real DOM/JS shapes.** Search cards
  (`article.item`) now yield rooms/area/floor (content-classified per chip,
  not positional — a card can carry an extra "Garaje incluido"-style chip in
  any slot), a description snippet, the agency name, and 1-3 photos — **at
  zero extra cost**, on the page every run already fetches. Detail-page
  enrichment (`enrichAgencyContact`, still off by default — see the 17x cost
  finding below) now actually extracts bathrooms, a richer floor label, the
  EPC energy-rating letter (rendered three different template-dependent ways
  across ES/IT/PT — all three handled), lat/lon (parsed from a Google Static
  Maps URL embedded in an inline `multimediaCarrousel` JS object), the
  agency's reference code, and the full gallery/description. The old
  JSON-LD/`#agency-data` path is kept only as an explicitly-labeled defensive
  fallback in case a future template adds structured data.
- **Fixed `location_city="Unknown"` — a sentinel string is worse than null.**
  It survived every null check and would have polluted any group-by a
  customer ran. `ResultRow.location_city`/`.actor/dataset_schema.json`
  changed from a required `str` (which forced the sentinel whenever the
  since-proven-fictional JSON-LD address block was absent, i.e. every row) to
  `str | None`; `location_city` is real (from the search-card title or the
  detail page's `.main-info__title-minor`) or explicitly `null`, never a
  placeholder.
- **Honest about what genuinely isn't there.** `agency_phone` and
  `listing_owner_type` (professional vs. private) are not exposed anywhere in
  idealista's static page markup on any of the three TLDs — phone
  specifically requires a separate authenticated AJAX call this Actor does
  not make — so both stay `null` unconditionally rather than guessed.
  `location_address` (a clean street, as opposed to the full title string),
  `location_province`, `listed_date`, and `updated_date` were the same:
  genuinely not resolvable from any page examined, so they stay `null` too.
  `src/models.py`, `.actor/dataset_schema.json`, `.actor/input_schema.json`,
  and the README (regenerated from `scaffold.json`, never hand-edited) were
  all updated to say this plainly instead of implying these fields sometimes
  populate.
- **Verification.** Full local `pytest` green against real-DOM-shaped
  fixtures (`tests/fixtures/search_{es,it,pt}.html` / `detail_{es,it,pt}
  .html`, replacing the fictional JSON-LD ones — a hypothetical JSON-LD
  fixture is kept separately, clearly labeled, only to exercise the
  defensive fallback branch). Independently, the rewritten parser was also
  run directly against the raw HTML from the live recon (not the fixtures)
  and every recovered field for one real ES listing
  (`https://www.idealista.com/inmueble/111433413/`) was hand-checked against
  that live URL: 2 rooms, 2 bathrooms, 102 m², floor "1ª planta interior con
  ascensor", energy rating D, agency "Gilmar Centro", reference "AG222680",
  coordinates (40.4232645, -3.7037263) — all correct. Repeated for one IT and
  one PT listing with the same result (IT's energy rating correctly resolves
  to `null` — that listing's own page genuinely carries no letter-grade
  marker, confirming this is honest, not a parser gap).
- `enrichAgencyContact` stays **off by default** — the 0.1.3 cost finding
  (~17x more residential-proxy transfer per row, $45.73 vs. $2.65/1,000 rows
  COGS) is unchanged by this fix; only what enrichment actually extracts
  changed, from nothing to the fields above.

### 0.1.3 — 2026-08-31

- **Pricing question resolved: two 100-row scale runs of `src.main` (the
  product, not recon), enrichment on and off, both against `madrid-madrid`/ES
  RESIDENTIAL.** `enrichAgencyContact=false` — run `ZnasXvRmHtHN8QLhx`, 100/100
  rows, settled $0.265274 total, 0.02832 GB residential = **0.29 MB/row =
  $2.65/1,000 COGS**. `enrichAgencyContact=true` — run `xfEy3LDqdVWn1G3SU`,
  44/100 rows (hit the per-country 600s internal deadline before reaching
  100\), settled $2.011924 total, 0.23165 GB residential = **5.39 MB/row =
  $45.73/1,000 COGS**, ~17x the unenriched path. Both settled stable after a
  5-minute re-check (no further drift) and both confirmed RESIDENTIAL via
  `scripts/os/run_usage.py`. At the current $0.005/row ($5.00/1,000) price,
  the default (now enrichment-off) path clears cost with a healthy margin
  (~$1.35/1,000 net after the 80% dev-share split); the enrichment path does
  not and is not priced as a paid tier (see next finding).
- **New finding, more important than the cost: enrichment currently extracts
  NOTHING.** 0 of the 44 enriched rows in `xfEy3LDqdVWn1G3SU`, and 0 of the 3
  rows in the original QA run `DEREkShO6HBt0DCFx`, carry any value in
  `agency_name`/`agency_phone`/`listing_owner_type`/`agent_reference_id`/
  `rooms`/`bathrooms`/`floor`/`energy_rating`/`location_address`/etc. — every
  field `_enrich_row` is supposed to populate is `null` on every real listing
  tested so far. `src/parser.py`'s `extract_detail_fields` looks for a
  `<script id="agency-data">` block and reuses the search-page JSON-LD walker
  on the detail page; `tests/fixtures/detail_es.html` is a 1.6 KB hand-written
  fixture that satisfies those exact selectors, so the unit tests pass while
  telling us nothing about the real page. This was never validated against a
  live idealista detail page. Net effect: turning enrichment on paid ~17x more
  residential-proxy transfer per row for the exact same fields you already
  had. Not fixed in this change (needs a real detail-page HTML capture and a
  parser rewrite against it — separate, larger piece of work) — flagged
  loudly instead: `enrichAgencyContact` now defaults to `false` in
  `src/models.py`/`.actor/input_schema.json`, with an updated field
  description warning it is experimental and non-functional. Un-shelved on
  the strength of the default (search-only) path, which is real, measured,
  and profitable — not on the enrichment path, which is neither.
- `.actor/pay_per_event.json`'s `result-row` description replaced with the
  measured basis above (run ids, MB/row, COGS/1,000 for both paths) — the
  previous text was provisional and derived from a recon run that wrote zero
  dataset rows.
- `tests/fixtures/input.qa.json` flipped `enrichAgencyContact` to `false` to
  match the new default and exercise the recommended, cost-safe path.

### 0.1.0 — 2026-08-26

- **Engine swap: curl-cffi → Camoufox.** curl-cffi through Apify RESIDENTIAL
  pinned `country_code=ES` got a flat DataDome 403 on every URL tried,
  including the bare site root (5/5 rotated exits, `docs/specs/idealista-
  property-scraper/CLOUD-RECON-RESULT.md`). A Camoufox recon probe with a
  same-context warm-up navigation (site root, then target, on the SAME Page
  object) cleared it: cloud run `YjJ5nU1187XfJum73` paginated 20 pages and
  parsed 571 rows for a SETTLED $0.9861 total (~$1.73/1,000 rows, 92% of it
  `PROXY_RESIDENTIAL_TRANSFER_GBYTES` — residential egress is the dominant
  cost here, not compute; an immediate read right after the run finished
  understated this 2.7x before the proxy billing line settled), with
  image/media/font/stylesheet requests aborted at the context level — see
  `src/browser.py`. New `src/browser.py` owns the Camoufox launch, resource
  blocking, and warm-up navigation; `src/scraper.py` and `src/main.py` were
  rewritten around it, replacing the curl-cffi session/backoff machinery with
  a per-country retry driver that relaunches the browser on a freshly
  rotated proxy exit for every attempt (never `geoip=True` — the documented
  `InvalidIP`/`InvalidProxy`/`LocaleError` crash mode from
  `vrbo-vacation-rentals-scraper/src/browser.py`; a static `locale=` per
  country is used instead). `src/recon_cloud.py` (the old curl-cffi
  throwaway recon module) is deleted — dead code once curl-cffi left the
  dependency set.
- **Fixed the real defect: the URL builder, not the target.** The pre-fix
  code built `/venta-viviendas/{location}/` straight from a bare city name
  (e.g. `"madrid"`), which 404s at idealista's own origin — proven
  side-by-side against the working `/venta-viviendas/madrid-madrid/` shape
  (municipality-province). `ActorInput.location` now documents that it must
  be idealista's own two-part slug; `src/config.py` never guesses or derives
  it (see that module's docstring for why: it's provably wrong for any
  municipality that isn't its own province capital). A 404 on search page 1
  now raises a distinct, non-retryable `InvalidLocationError` instead of
  being treated as a possible block — the retry driver stops immediately
  rather than burning attempts on an unwinnable request.
- New `AntiBotChallengeError` (non-200 status / bot-detection interstitial /
  geo\_guard mismatch on a search page) drives the per-country retry;
  `EmptySearchResultError` (a genuinely empty, non-blocked page 1) still
  fails loud without retrying.
- `.actor/Dockerfile` rebuilt on `apify/actor-python-playwright:3.11` with a
  `python -m camoufox fetch` build step; `CMD` is `python -m src.main`
  (the real entrypoint) — there is no recon module on the entrypoint path.
  `pyproject.toml` swaps `curl-cffi` for `camoufox[geoip]` + `playwright`;
  memory bumped to 2048–4096 MB (`minMemoryMbytes`/`maxMemoryMbytes`/
  `defaultMemoryMbytes`) to match the fleet's other Camoufox actors.
  `pay_per_event.json`'s `actor-start` set to $0.20 (fleet-standard for
  Camoufox actors — vrbo/glassdoor/booking-hotels all use this), `result-row`
  set to $0.005 — priced on this Actor's own measured COGS floor above, not
  matched to curl-cffi comparables (rightmove $0.002/row, funda $0.0012/row)
  whose cost structure is structurally lower.
- COGS fix: detail-page enrichment previously re-navigated to the ~30-item
  SEARCH page as the warm-up before EVERY detail-page fetch on that page (up
  to 30x redundant re-fetches of the same page per page of results). New
  `_WarmupCursor` (`src/scraper.py`) applies the same proven "warm on the
  immediately-preceding fetched URL" principle already used for search
  pagination to detail-page enrichment too — only the first detail fetch on
  a page warms up via the search page; every fetch after that warms up via
  the previous detail page. `browser.open_page` now returns a
  `ResourceBlockStats` counter so `main.py` logs allowed-vs-blocked request
  counts per attempt, proving the abort route is actually firing.
- Verified locally: `apify run` against `tests/fixtures/input.qa.json`
  (ES, `location=madrid-madrid`) returned 3 real, live Madrid listings via
  Apify RESIDENTIAL/ES through Camoufox — confirms the URL fix and engine
  both work end to end before any cloud spend.
- **Cloud QA (post-push, correct pricing live): run `DEREkShO6HBt0DCFx`,
  build `nF79ZreRPztyHeJoZ` (0.1.2), SUCCEEDED.** Attempt 1 hit a 403 on
  search page 1 (soft block) and the retry driver relaunched the browser on
  a freshly rotated proxy exit; attempt 2 succeeded. 3/3 real Madrid
  listings returned (same rows as the local run). `chargedEventCounts`:
  `actor-start=1` ($0.20), `result-row=3` ($0.005 each = $0.015) — both PPE
  events fired correctly (count reporting itself lags ~1 minute after
  `SUCCEEDED`, same settlement-lag pattern as `usageTotalUsd`; re-read
  confirmed 3, not the immediate 0). Settled run cost: `usageTotalUsd`
  $0.2225, of which `PROXY_RESIDENTIAL_TRANSFER_GBYTES` = $0.1980 (89%).
  This 3-row QA run's $/row is NOT representative of production economics —
  a `maxResultsPerCountry=3` fixture amortizes one full search-page fetch
  (and one wasted, blocked retry attempt) over only 3 rows instead of the
  \~30 a real page delivers (`reference-actor-cost-measurement`); the 571-row
  recon run's $1.73/1,000 (search-only, no enrichment) remains the
  representative cost floor. Production enrichment adds real per-row cost
  on top of that floor that hasn't been measured at production scale;
  result-row is priced with a margin above the known floor, and should be
  revisited once real run volume gives a believable at-scale sample.
- Tests rewritten for the new engine (`tests/test_browser.py` new,
  `tests/test_scraper.py`/`tests/test_main.py` rewritten): the 404-shape
  regression asserts both directions (`madrid-madrid` produces the working
  URL; it never collapses back to the old `madrid`-only 404 shape).

### 0.0.1 — 2026-08-11

- Scaffolded: `.actor/` (actor.json, input/output/dataset schemas, PPE,
  Dockerfile), `src/models.py` fully implemented (`ActorInput` + `ResultRow`
  per spec, REQ-2 search\_urls-or-countries+location validator), placeholder
  `src/main.py` (boots, validates input, pushes one `{"placeholder": true,
  ...}` row, exits — no real crawler), tests, README skeleton.
- Real implementation (`src/config.py`, `src/geo_guard.py`, `src/parser.py`,
  `src/scraper.py`) is pending: per-TLD wire format, pagination pattern, and
  geo-splash locale markers are all `needs-cloud-recon` — see
  `docs/specs/idealista-property-scraper/spec.md` ("Assumptions", "Risks")
  and `docs/specs/idealista-property-scraper/tasks.md` T02–T12.