# Changelog of Bayut UAE Real Estate Scraper (`devilscrapes/bayut-uae-real-estate`) Actor

- **URL**: https://apify.com/devilscrapes/bayut-uae-real-estate/changelog.md
- **Full Actor documentation**: https://apify.com/devilscrapes/bayut-uae-real-estate.md

## Bayut UAE Real Estate Scraper — Changelog

### 0.1.3 — 2026-09-19

- **Un-shelve fix, billing defect (the reason this Actor was SHELVED).**
  `actor-start` used to charge at boot, before input validation and before
  any proxy/browser existed. Run `Hy8dPuLBeAhGBhXKP` (100 rows requested)
  delivered 0 rows and still billed the customer $0.30 — the exact
  `vrbo-vacation-rentals-scraper` delisting shape. The fee now fires from
  `src/main.py`'s `_StartFee.charge_once()`, called from `_flush` on the
  first batch that actually reaches the dataset — ported verbatim from
  `upwork-job-leads-scraper`/`tiktok-ad-library-scraper`. `verify_charge_order.py`
  now reports `no-start-charge` (was `CHARGE-FIRST`).
- **Un-shelve fix, the actual cause of the zero-row delivery.** That same
  run's two retry attempts were: attempt 1, a real Bayut anti-bot challenge
  (target-side); attempt 2, `Page.content: Unable to retrieve content
  because the page is navigating and changing the content` — Playwright's
  own wording for a still-in-flight client-side redirect racing the read,
  **not** a target block. That wasted the entire retry budget on our own
  bug instead of on Bayut. `src/browser.py`'s `fetch_page` now reads
  content through `_read_content_with_retry`, ported verbatim from
  `glassdoor-reviews-scraper` (2026-08-25 cloud QA run
  `r0DmkqqyFv2ng8vhH`): retries a bounded number of times on that exact
  race marker, re-raises immediately on any other `PWError`.
- Reachability and unit economics were already retired before this fix
  (8/8 SUCCEEDED across 0.1.4-0.1.6; $2.93/1,000 settled cost against
  $0.008/row + $8.30/1,000 gross — see `scripts/publish_actors.py`'s
  retired shelve note) — this release clears the two remaining defects.
- 5 new regression tests (`tests/test_browser.py`'s
  `_read_content_with_retry` cases; `tests/test_main.py`'s `_StartFee`/
  `_drain_rows` cases) lock both fixes: a zero-row run, a fully-blocked
  run, and a duplicate-listing run all charge nothing beyond what was
  actually delivered. 71/71 tests pass, ruff clean, pyright clean.

### 0.1.2 — 2026-09-18

- **Removed the `enrichWithDetails` option and the detail-page navigation path
  it gated — before this Actor's first public monetized release.** 0.1.1 made
  the option opt-in and off-by-default, but under Pay-Per-Event the developer
  (not the customer) pays the platform's underlying compute cost of every
  customer run, including opted-in ones. At $62/1,000 rows against ~$8/1,000
  of revenue, a customer-flippable switch that loses money on every use is a
  hazard a monetized Actor cannot carry — and a monetized Actor cannot be
  unpublished (`cannot-unpublish-paid-actor`), so this had to be fixed before
  the flip, not after.
- **Removed `permit_number` and `amenities` from `ResultRow`** — both fields
  were sourced exclusively from the now-removed detail-page fetch. We would
  rather not sell a column than sell one we cannot fill.
- **Removed `listed_date`.** It genuinely only ever extracted from the detail
  page (`//span[@aria-label="Reactivated date"]`) — confirmed absent from the
  search-page JSON-LD and HTML (checked against the live-shape fixtures
  captured 2026-08-26; no date field anywhere in `mainEntity`). There is no
  cost-free way to source it, so it is gone rather than shipped null.
- **`price_frequency` and `agent_phone` are unchanged** — both are genuine
  search-page fields, sourced from JSON-LD, with no dependency on the removed
  detail-page path. `price_frequency` is documented `null` for sale listings
  by design. `agent_phone` requires `includeAgentPhone=true` (also by
  design — most runs don't ask for it) and is confirmed working: a same-day
  cloud measurement (build 0.1.4, run `rKEzSa3zoa2iBvqoW`, both flags on)
  showed `agent_phone` **8/8**. `field_fill_audit.py` flags it as always-empty
  only because the run it audited did not set `includeAgentPhone=true` — the
  same non-defect shape as `price_frequency`.
- `scraper.py` no longer threads `enrich_with_details` through
  `SearchRunConfig`, and `_enrich_row`/`_WarmupCursor` are deleted; every run
  now performs exactly one `fetch_page` call per search page and zero
  per-row browser navigations. `parser.py` drops
  `enrich_with_detail`/`extract_permit_number`/`extract_listed_date`/
  `extract_amenities` and their XPath constants.
- README, `.actor/input_schema.json`, `.actor/dataset_schema.json`, and
  `.actor/actor.json`/`pyproject.toml` descriptions no longer advertise the
  removed option or fields. Pricing (`$0.008`/row, `$8.30`/1,000) is
  unchanged — COGS is down, not the price; that is a separate call for a
  human to make.
- 62/62 tests pass (down from 75 — enrichment-only test cases removed with
  the code they covered), ruff clean, pyright clean on `src/`.

### 0.1.1 — 2026-09-13

- **COGS fix — the reason this Actor was SHELVED.** Unconditional per-row
  detail-page enrichment (`permit_number`/`listed_date`/`amenities`) drove
  the Actor to $62/1,000 rows against ~$8/1,000 of revenue (6.92 MB/row of
  residential-proxy transfer for one extra Camoufox navigation per listing;
  `scripts/publish_actors.py`'s shelve note). `enrichWithDetails` is now a
  plain input checkbox, **off by default** (`models.py`, `input_schema.json`,
  `scraper.py`'s `SearchRunConfig.enrich_with_details`) — the default run
  never fetches a detail page, matching the ~2.08 MB/row search-only floor
  the Camoufox recon already measured (420 rows, $0.871969 settled).
- Also a product finding, not just a cost one: across 28 real enriched rows
  pulled from two prior production/QA runs, `listed_date` extracted
  correctly on all 28, but `permit_number` and `amenities` came back
  null/empty on all 28. Enrichment is kept available (opt-in) rather than
  removed, but the README and input-field description say plainly that two
  of its three fields have not been observed to work.
- README rewritten to describe the new default: "What this scrapes",
  Input/Output tables, the example output row, Pricing, Limitations, and a
  new FAQ entry all reflect a search-only default with enrichment as an
  explicit, costlier opt-in.
- New regression tests lock the gate: the default `SearchRunConfig` never
  awaits `_enrich_row`; setting `enrich_with_details=True` does;
  `_search_run_config` threads `ActorInput.enrich_with_details` through
  correctly. 74/74 tests pass, ruff clean, `uv run pyright src` clean.
- Still SHELVED pending a human publish decision — this fix does not
  un-shelve the Actor in `scripts/publish_actors.py`; see the fixer's
  report for the measured $/1,000 verdict.

### 0.1.0 — 2026-08-26

- Un-shelved: real Camoufox-based crawler replaces the placeholder scaffold.
  `docs/specs/bayut-uae-real-estate/CLOUD-RECON-RESULT.md`'s "JS-execution
  wall, curl-cffi insufficient" verdict stands, but Camoufox with a
  same-context warm-up navigation (`src/browser.py`, carried forward from
  `src/recon_camoufox.py`, proven live 2026-08-26: 20/20 pages, 420 rows)
  clears it.
- `src/parser.py`: two-source extraction — JSON-LD `@graph[0].mainEntity.
  itemListElement` (primary, all search-page fields) zipped by position with
  `aria-label`-anchored HTML cards (`verified`, `location_hierarchy`
  fallback). Detail-page enrichment (`permit_number`, `listed_date`,
  `amenities`) runs for every row by default — the spec's own differentiator
  over 22+ competitor Actors.
- `src/scraper.py`: pagination, per-listing fault isolation, bed/price
  filters, dedup, and Page recycling every 3 fetches (search+detail
  combined) — a single long-lived Page crashes Firefox after ~6 real
  24-item pages (Camoufox stability limit, not a Bayut block).
- `src/main.py`: retry driver (fresh proxy exit per attempt), PPE charges
  (`actor-start` $0.01, `result-row` $0.002), empty-vs-block distinction
  (a genuine zero-match search SUCCEEDS; only an exhausted-retries block
  fails loud).
- Memory bumped 256→2048 MB (Camoufox needs it under Apify's
  `LIMITED_PERMISSIONS` container). Proxy default pinned to
  `RESIDENTIAL`/`country_code=AE`.
- Icon replaced: the scaffold's generic "BY" text monogram is now a
  recreation of Bayut's real house-in-magnifying-glass mark.
- Scope cut: `polygon` search input is not implemented (query-param
  encoding was never confirmed) — a polygon-only run fails loud with a
  clear message rather than guessing a URL shape. `searchUrls` and
  `purpose`+`location` are the supported v0.1 paths.

### 0.0.1 — 2026-07-20

- Scaffolded: `.actor/` (actor.json, input/output/dataset schemas, PPE,
  Dockerfile), `src/models.py` fully implemented (`ActorInput` + `ResultRow`
  per spec, REQ-2 XOR validator), placeholder `src/main.py` (boots, validates
  input, pushes one `{"placeholder": true, ...}` row, exits — no real
  crawler), tests, README skeleton.
- Real implementation (`src/client.py`, `src/parser.py`) is blocked on a
  cloud-recon pass confirming Bayut's live JSON wire format
  (`__NEXT_DATA__` vs. an Algolia search endpoint) behind a genuine
  RESIDENTIAL-proxy platform run — see
  `docs/specs/bayut-uae-real-estate/spec.md` ("Assumptions", "Risks") and
  the `importyeti-alternative-scraper` T14 precedent.