# Changelog of Crexi Real Estate Scraper (`crawlerbros/crexi-real-estate-scraper`) Actor

- **URL**: https://apify.com/crawlerbros/crexi-real-estate-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/crawlerbros/crexi-real-estate-scraper.md

## Changelog

All notable changes to the Crexi Real Estate Scraper will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

### \[1.4.0] - 2026-08-07

#### Changed

- **Input schema simplified to filters-only**: removed `runTimeoutSecs` (internal
  budget, now hardcoded) and `injectSavedCookies` (debug knob for the browser
  fallback, now always off). `proxy` is no longer required and has no default —
  the primary path needs no proxy at all. Users now only pick: `maxProperties`,
  `scrapeDetails`, `propertyTypes`, `locations`, `minPrice`, `maxPrice`.
- `maxProperties` default lowered to 100; `scrapeDetails` default false.
- README updated (input table, examples, FAQ).

### \[1.3.0] - 2026-08-06

#### Changed

- **Primary data path is now the public search API** (`api.crexi.com/assets/search`).
  Crexi moved its frontend search behind session-gated pages in Aug 2026 — anonymous
  browsers no longer receive the `/assets/search` call, so the browser flow returned
  zero properties on every run. The direct API needs no cookies, no browser, and no
  proxy, and returns data in seconds.
- Request schema reverse-engineered from the frontend bundles: `{ brokerIds, count,
  includeUnpriced, mlScenario: "search-properties", offset, sortDirection, sortOrder,
  types?, states?, askingPriceMin/Max? }`; response `{ data, totalCount }`.
- Record mapping rewritten for the new item shape (`locations[]` with
  `state{code,name}`, `thumbnailUrl`, `urlSlug`, `types[]`, `askingPrice`,
  `squareFootage`).
- `scrapeDetails` now enriches via the public detail API (`GET /assets/{id}`):
  `details` key/value map, full `description`, `net_operating_income`, `subtypes`.
- Strict client-side filter confirmation: the API's `states`/`types` filters are
  inclusive, so records are additionally verified to match the requested state, type,
  and price range (unpriced listings are excluded when a price filter is set).
- Search URL strategy updated for the new `/search/<type-slug>/<state>` format with
  legacy `/properties/<state>` fallback (used only by the browser fallback path).
- Browser path (Playwright + residential proxy) kept as a fallback when the API is
  unreachable; saved-cookie injection is now opt-in (`injectSavedCookies`).

#### Added

- `curl_cffi` dependency for the direct API client.
- Unit tests for the API request builder, record mapping, and detail merging.

### \[1.0.0] - 2025-10-30

#### Added

- Initial release of Crexi Real Estate Scraper
- Automated scraping of commercial real estate listings from Crexi.com
- Property data extraction including:
  - Basic information (name, type, location, address)
  - Financial data (price, lease rate, investment metrics)
  - Physical details (square footage, lot size, specifications)
  - Media (images, documents)
  - Agent information
  - Property descriptions and highlights
- Pagination handling for comprehensive data collection
- Rate limiting with configurable delays
- HTML debugging feature that saves page content for selector analysis
- Browser automation using Playwright
- Anti-detection measures (user agent, header customization)
- Multiple selector strategies with fallback options
- Detailed property page scraping (optional)
- Data validation and cleaning
- Duplicate detection and prevention
- Configurable input parameters:
  - `maxProperties`: Maximum number of properties to scrape
  - `scrapeDetails`: Toggle for detailed page scraping
  - `propertyTypes`: Filter by property types
  - `locations`: Filter by locations
  - `minPrice` / `maxPrice`: Price range filters
  - `rateLimitDelay`: Configurable delay between requests
- Comprehensive error handling and logging
- Local testing script (`test_local.py`)
- Output visualization script (`view_output.py`)
- Docker support with Dockerfile
- Apify platform integration with:
  - Actor configuration (`.actor/actor.json`)
  - Input schema (`.actor/input_schema.json`)
  - Dataset views
- Comprehensive documentation:
  - README.md with features and usage
  - IMPLEMENTATION\_GUIDE.md with technical details
  - PROJECT\_STRUCTURE.md with file organization
  - QUICK\_START.md for rapid onboarding
- Python package structure with proper imports
- Git ignore configuration

#### Features

##### Data Extraction

- 20+ data fields per property
- Support for both listing and detail page extraction
- Flexible selector strategies to handle various page layouts
- Automatic URL normalization and validation

##### Performance

- Configurable rate limiting (default: 2 seconds)
- Efficient pagination handling
- Option to skip detail pages for faster scraping
- Memory-efficient processing

##### Reliability

- Multiple selector fallbacks
- Comprehensive error handling
- Graceful degradation when fields are missing
- Deduplication to prevent duplicate entries
- Anti-detection browser configuration

##### Debugging

- Automatic HTML saving for analysis
- Detailed logging at each step
- Debug mode with HTML snapshots
- Selector discovery helpers in logs

##### Testing

- Local test script for development
- Output visualization tool
- Sample input configurations
- HTML analysis workflow

#### Technical Details

- Python 3.12 support
- Playwright 1.40.0 for browser automation
- BeautifulSoup 4.12.0 for HTML parsing
- Apify SDK 2.1.0+ integration
- Async/await pattern for efficiency
- Type hints for better code quality

#### Documentation

- Complete README with usage examples
- Implementation guide for developers
- Project structure documentation
- Quick start guide
- Changelog for version tracking

#### Known Limitations

- Requires internet connection
- Subject to Crexi website structure changes
- Some fields may not be available for all properties
- Rate limiting required to avoid blocking
- Respects robots.txt and terms of service

#### Future Enhancements (Planned)

- \[ ] Add support for saved searches
- \[ ] Implement proxy rotation
- \[ ] Add CSV export option
- \[ ] Create scheduled run templates
- \[ ] Add data validation rules
- \[ ] Implement webhook notifications
- \[ ] Add support for custom filters
- \[ ] Create dashboard for monitoring
- \[ ] Add data enrichment features
- \[ ] Implement incremental updates

### \[Unreleased]

#### Planned

- Enhanced error recovery mechanisms
- More granular logging levels
- Performance optimizations
- Additional data fields
- API integration options

***

### Version History

#### Version Numbering

- **Major version (X.0.0)**: Breaking changes or major feature additions
- **Minor version (1.X.0)**: New features, backward compatible
- **Patch version (1.0.X)**: Bug fixes, minor improvements

#### Release Notes Format

- **Added**: New features
- **Changed**: Changes to existing functionality
- **Deprecated**: Features that will be removed
- **Removed**: Removed features
- **Fixed**: Bug fixes
- **Security**: Security improvements

***

*Note: This changelog will be updated with each release. For the latest changes, see the commit history.*