Scrape normalized US Redfin listings — for-sale, sold, and pending — with full MLS/geo field coverage and reliable pagination past Redfin's 350-result cap via price-bucket bisection, deduped by property ID.
Fix: customers charged for runs that produced zero rows. Root cause —
the account's default proxy group (BUYPROXIES94952, datacenter) gets a
hard 405 from Redfin's edge on every single search-page request, live-
probed from this machine across all four BROWSER_PROFILES. Confirmed the
405 was our own datacenter IP, not a wrong HTTP verb — client.py
issues a genuine GET throughout, verified against curl_cffi's source.
src/models.py's proxy_configuration field validator (spec REQ-11) now
defaults apifyProxyGroups to ["RESIDENTIAL"] instead of forbidding it
— the original prohibition was written on the FREE plan against an
availableCount:0 cardinality misread (see memory
reference-residential-is-entitled); this account has been on STARTER
since 2026-08-20 and RESIDENTIAL clears with a clean 200.
.actor/input_schema.json default/prefill updated to match.
Fix (found while verifying the proxy fix): every row was silently
dropped as "unparseable" — a second, independent defect masked by the
405.src/parser.py::home_to_result_row required propertyId to be a
JSON string; the live gis payload sends it (and listingId) as a JSON
number. New _stringify_or_none() helper stringifies numeric
IDs/enum codes faithfully instead of dropping them. Also applies to
searchStatus/listingType/propertyType/uiPropertyType, which are
small integer enum codes in the live payload, not the semantic strings
("Active", "for-sale", "Single Family Residential") spec.md and
tests/fixtures/gis_response_sf.json originally — and never actually
live-verified — assumed. Those four fields now surface as raw numeric-
code strings rather than a guessed semantic label (guessing would
fabricate a value, forbidden by REQ-8).
Confirmed (not changed): the REQ-13 "zero rows across all entries fails
loud" guard in src/main.py::main() was already correct in this build
(SystemExit(1), never a silent SUCCEEDED) — the 2026-08-15 build that
shipped a false SUCCEEDED-with-zero-rows predates it. Regression test
tests/test_main.py::test_zero_rows_across_all_entries_raises_systemexit
already pins this and continues to pass.
Verified end-to-end: apify run against the real RESIDENTIAL proxy
(tests/fixtures/input.qa.json) now scrapes 8/8 real Redfin listings,
exit code 0 — was 0 rows / exit 91 before this fix.
See docs/specs/redfin-property-listings-scraper/notes.md for the full
probe table and root-cause writeup.
0.1 — 2026-08-12
Real crawler implemented end to end: src/resolver.py (pure RSC gis
query-string extraction tolerating single-/double-escaped /,
ExpectedGeo/ResolvedRegion DTOs, two-signal passes_geo_guard()
for REQ-3's geo-splash defense), src/client.py (the only module
importing curl_cffi — RedfinSession with browser-profile rotation,
408/429/503/network-error retry with 2s→30s capped exponential
backoff, session_id rotation, {}&&-prefix-stripping + resultCode
validation for the gis envelope), src/paginator.py (price-bucket
bisection past Redfin's 350-home server cap with mpt=99, propertyId
dedup via a shared seen_ids set, per-bucket fault isolation),
src/parser.py (pure homes[] dict → ResultRow mapping, never
fabricating a value), and src/main.py (SDK wiring: actor-start/
result-row PPE charging, per-entry geo-guard rotate-retry-once-then-
skip, per-entry fault isolation, REQ-13 fail-loud on zero rows).
Live-confirmed 2026-08-12 (real network fetches against redfin.com):
the RSC regex, the {}&& envelope prefix, the 350-home cap, and the
mpt=99 bisection requirement all match spec.md's wire-format claims
verbatim. Additionally confirmed empirically during implementation
that status="sold" maps to sold_within_days=90 (with sf absent
from the query) — spec.md's Open Questions item 1 for sold is now
resolved. The pending status mapping remains an unconfirmed
best-effort placeholder (STATUS_PARAMS["pending"], see the inline
comment in paginator.py) after several live probes returned either
the for-sale-shaped default or a bot-challenge page — paginator/parser
stay permissive so a wrong guess degrades gracefully, never crashes.
Also discovered live that soldDate/openHouseStart/openHouseEnd
arrive as epoch-ms integers, not the ISO YYYY-MM-DD strings spec.md
assumed — parser._epoch_ms_or_str_to_iso() handles both shapes.
62 unit/integration tests (tests/test_resolver.py,
tests/test_client.py, tests/test_paginator.py,
tests/test_parser.py, tests/test_main.py) — all pure/mocked, no
live network in CI. ruff check clean.
0.0.1 — 2026-08-12
Scaffolded: .actor/ (actor.json, input/output/dataset schemas, PPE,
Dockerfile), src/models.py fully implemented (ActorInput + ResultRow
per spec, search_urls-or-region-triple validator, RESIDENTIAL-proxy
rejection), placeholder src/main.py (boots, validates input, pushes one
{"placeholder": true, ...} row, exits — no real crawler), tests, README
skeleton.
Real implementation (src/resolver.py, src/client.py,
src/paginator.py, src/parser.py) is pending — RSC gis extraction,
price-bucket bisection past the 350-home cap, and the two-signal
geo-splash guard are all live-confirmed wire format per
docs/specs/redfin-property-listings-scraper/spec.md ("Assumptions") but
not yet wired up; see docs/specs/redfin-property-listings-scraper/tasks.md
T02–T12.
