Zillow Real Estate Scraper

2023-10-18 (v0.0.37)

Fixes

• Fixed issue causing Error: Cannot find module 'functions-have-names' and took steps to stop this from happening again in the future. (This issue was caused by a bugged minor release of a package that we depend on, and our package-lock.json file wasn't committed alongside our code, so updates of dependencies weren't properly tested in our pipeline.)

2023-06-28 (v0.0.27)

Features

• added monthly homeowner association (HOA) fee to output (monthlyHoaFee)

2023-06-27 (v0.0.25)

Fixes

• Detect new variant of Zillow captcha (this was causing zero results on search)

Changes

• Property detail URLs (zillow.com/homedetails/.../123456_zpid/) are now also supported as startUrls. This is equivalent to providing the numeric part as a zpid.

2023-06-21 (v0.0.24)

Changes

• Lot size is now also available in simple input.

2023-05-21

Fixes

• Update queryId detection to another Zillow code change.

2023-02-03

Changes

• Use RESIDENTIAL proxy group by default for better unblocking and increase retries

Fixes

• Hotfixed scraper not starting because it could not retrieve queryId

2023-01-13

Features:

• Added timeOnZillow, pageViewCount, favoriteCount and chanceToSellFaster fields to the output

2022-01-04

Fix:

• Handled issue where "For Rent" listings were not being scraped

2022-12-19

Features:

• Added zestimate, rentZestimate, and listedBy fields to the output
• Moved isListedByOwner field to listedBy object
• listedBy object includes the following fields name, email, phone, profileUrl, and isListedByOwner

2022-03-17

Features:

• Refactor the scraping and map splitting
• Add support for US zipcodes
• Faster scraping
• Supports city wide scraping

Fix:

• Map zoom for street level
• Query state for URLs

2022-02-11

Features:

• Better splitting of big areas
• Better filtering of results
• Better logging

Fix:

• Map zoom on splits
• Handle page timeout in 120s instead of 1h
• Retrying of empty results
• Paginating of results
• Retrying of pages without zpid
• Unecessarily retrying on invalid listings

2022-02-06

Fix:

• Photos

2022-01-07

Features:

• Add pagination search inspecting listing results (more items scraped while testing)
• Use pagination search for all pages (with properties <= 500, > 500 ) for more results

Bug fixes:

• Fix query states fetch request (map results weren't loaded properly)
• Remove residential proxies recommendation
• Use map splitting only for results > 500

Dev:

• Refactor code (split main function, handlePageFuntion)
• Update to SDK v2.2
• Replace header generator for user-agent with SDK built-in fingerprints support
• Handle most of the tslint warnings

2021-12-01

Bug fixes:

• Handle request blocking by retrying requests with 0 results found
• Increase number of dataset results by forcing map splitting for results < 500

2021-08-03

Feature:

• Update SDK to latest
• Translation of searchQueryState to search query
• Use header generator for user-agent

Bug fixes:

• Workaround too many captchas

2021-07-21

Feature:

• Updated SDK
• Min / max dates for listings
• Debug log

Changes:

• Strictly filter everything if Type isn't "All"
• Better management of max items
• Handle interstitial

Bug fixes:

• FSBO listings
• Getting query id
• Properly retiring browser instance
• "All" setting with search term

2021-01-30

Features:

• Completely refactored code
• Support for external urls for start urls
• Can accept search urls containing searchQueryState with specific filters
• Added improved extendOutputFunction and extendScraperFunction
• Additional parameters that can be provided
• Improved initial search
• Added ability for sold listings
• More consistent output for number of items
• Hint for residentials, works much better than datacenter proxies
• Filter for data not containing zipd (non-listing)

Fixes:

• Fixed infinite loop while scanning for houses
• Lint and code reorganization
• Retries failing detail urls
• Require the use of proxies
• Better headers to match the website requests through fetch