Scraping American real estate market for research

Quick summary

Meet Francis Hodgens, a talented researcher from Boston College who received a grant to look into property tax in the Massachusetts area. He’s currently working on a research project that requires a great deal of data collection from a real estate platform that is traditionally restrictive when it comes to web scraping.

The first research goal was to scrape a US real estate platform for various data on houses. The student chose a popular real estate website for this task since this website provided the biggest and most versatile range of data on the American housing market. After facing many issues with existing APIs, Fran needed something better, so he decided to give Apify a try. This is the moment when one of our real estate scrapers saved the day. Apify’s tool handled every request with ease and has provided more time for Fran to perform a better, more detailed analysis as a researcher.

Full story and the challenge

The real estate website Fran wanted to scrape was a great resource for the purpose of data collection, since it enabled him to look at house market data on a granular level. Fran first tried scraping using the official website API; that didn't go too well. Then, since he had a background in programming and web scraping, Fran spent some of that precious research time creating a couple of scraping programs. Unfortunately, no matter how good they were, his programs didn’t get him too far in research. He finally searched for an online real estate web scraper - and came across the real estate APIs in Apify Store. After signing up, Fran found that the convenience of a free trial made his research more than affordable.

There were sometimes minor issues with one of the scrapers, but Apify support usually got back to him within minutes to help. He made a special request to try out residential proxies for a couple of weeks and we enabled those for him to help with his studies. One bigger issue with the actor that Francis ran into was that at some point the dates did not work properly. After notifying the support team about it, the problem was fixed within a couple of days.

He eventually managed to scrape all home sales in his selected area from January 2021 to June 2021, and extracted 3,000 rows of data with properties like size of house, year of construction, etc. The exported data was then integrated into Tableau - a Business Intelligence tool for data visualization. Then he experimented with it to see if there was any relationship between what the house was sold for, the sale price, and the state evaluation of the house. By collecting and evaluating all that data, the student was able to analyze housing prices and conditions of all 22 communities outside of Boston. He then counted property tax for every single house and was able to identify a negative correlation between sale price and tax assessment/sale ratio that constitutes the core of his research findings.

Web scraping for students is the future of research

Although Fran was a pioneer in using Apify tools at his university, he has found our scapers to be the most effective option for web scraping for academic purposes. As someone with only a few months' worth of experience with web scraping, Fran has also found Apify's detailed blog posts and very friendly customer support invaluable in his research. But don't just take our word for it, read more of Francis’s reviews here, here and here.

We have more exciting research-related case studies in stock for you. Read about how web scraping provided a great basis for undergraduate data analysts to combat the spread of clickbait and fake news in the media. If you need any more examples of how web scraping can be used for academic work, head over to our Research and Education page.

Want to scrape a real estate listings website, but can't find a solution? Let us know, we can build any customized real estate scraper from scratch!