# HotPads Rental Listings Scraper (`x.com/hotpads-listings-scraper`) Actor

Scrape rental listings from HotPads with anti-detect Camoufox browser, extracting structured data from server-side state blobs. Includes price, bedrooms, amenities, pet policies, and more.

- **URL**: https://apify.com/x.com/hotpads-listings-scraper.md
- **Developed by:** [Girma Wakeyo](https://apify.com/x.com) (community)
- **Categories:** Agents, Automation, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## 🏠 HotPads Rental Listings Scraper

**HotPads Rental Listings Scraper** is a production-grade Apify Actor that extracts structured rental listing data from [HotPads](https://hotpads.com) using an anti-detect Camoufox browser with geo-matched fingerprints. It reads data directly from HotPads' server-side state blobs (`window.__PRELOADED_STATE__`) instead of fragile CSS selectors, providing more stable and richer data extraction.

Perfect for rental market analysts, real estate investors, property managers, and AI agents that need reliable access to rental listing data without manual browsing.

### ✨ Key Features

- **🦊 Anti-Detect Camoufox Browser:** Uses Camoufox (stealthy Firefox fork) with geo-matched fingerprints built inside the browser launch lifecycle — not at module scope. Each browser session gets a fingerprint derived from its actual exit IP, defeating PerimeterX bot detection.
- **📊 Server-Side State Extraction:** Reads directly from `window.__PRELOADED_STATE__` instead of scraping the DOM. This is more stable across site redesigns and contains richer data (price drops, popularity counters, school ratings) that never renders on screen.
- **🔍 Smart Block Detection:** Detects PerimeterX challenges, 403/429 errors, and soft-blocks (missing state scripts). Blocks fail loudly with retries instead of silently returning empty datasets.
- **🐾 Pet Policy Data:** Extracts detailed pet policies including cat/dog allowances, size limits, and deposits — data unique to HotPads' state blob.
- **🏫 School Information:** Pulls nearby school data with ratings, grade ranges, and distances from the state blob.
- **📈 Price Drop Tracking:** Captures price drop amounts and flags from the state blob — data the page never renders.
- **🔧 AI Agent Optimized:** Input schema descriptions encode site limits and cost tradeoffs so AI agents can plan runs correctly without human supervision.

### 🎯 Who is this for?

- **Rental Market Analysts:** Track pricing trends, availability, and pet policies across neighborhoods.
- **Real Estate Investors:** Identify price drops and investment opportunities.
- **Property Managers:** Monitor competitor listings and pricing.
- **AI Agents & Automation:** Reliable, structured data extraction for autonomous workflows.
- **Data Scientists:** Clean, JSON-formatted rental data for analysis and modeling.

***

### 🚀 How to use

#### Step 1: Get your HotPads search URL

1. Go to [hotpads.com](https://hotpads.com)
2. Search for rentals (city, neighborhood, filters)
3. Copy the URL from your browser address bar
   - Example: `https://hotpads.com/san-francisco-ca/apartments-for-rent?beds=1&maxPrice=3000`

#### Step 2: Configure the Actor

Paste the URL into the **Start URLs** field. Adjust settings:

- **Max Pages:** How many pagination pages to follow (each page = ~40 listings). HotPads hard-caps at 40 pages (~1,600 listings per search).
- **Include Details:** Enable to visit each listing's detail page for full descriptions and pet policies (slower, more compute units).
- **Max Concurrency:** 2-3 is safe; higher risks blocks even with residential proxies.

#### Step 3: Run and download

Click **Start** and wait for the run to finish. Download your dataset in JSON, CSV, HTML, or Excel format from the **Output** tab.

***

### 📥 Input

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| **Start URLs** | array | `hotpads.com/san-francisco-ca/apartments-for-rent` | HotPads search result URLs. Each URL should be a search page from hotpads.com. |
| **Max Pages** | integer | `40` | Pagination pages per URL. HotPads hard-caps at 40 (~1,600 listings). |
| **Include Details** | boolean | `false` | Visit detail pages for full descriptions and pet policies (slower, more CU). |
| **Max Concurrency** | integer | `2` | Simultaneous pages. 2-3 is safe; 5+ risks blocks. |
| **Proxy Configuration** | object | Residential Proxies | Anti-bot proxy settings. Residential strongly recommended. |

***

### 📤 Output

Each listing is a structured JSON record with explicit `null` values for missing data (not omitted keys). This ensures downstream tools can distinguish "Actor doesn't return this field" from "data wasn't available."

```json
{
    "listing_id": "12345678",
    "url": "https://hotpads.com/san-francisco-ca/apartments-for-rent/12345678",
    "source_url": "https://hotpads.com/san-francisco-ca/apartments-for-rent",
    "title": "Modern 1BR in SoMa",
    "property_type": "Apartment",
    "address": "123 Main St, San Francisco, CA 94105",
    "city": "San Francisco",
    "state": "CA",
    "zip_code": "94105",
    "neighborhood": "SoMa",
    "latitude": 37.7749,
    "longitude": -122.4194,
    "price": 2800,
    "price_min": 2800,
    "price_max": null,
    "price_drop": 200,
    "has_price_dropped": true,
    "bedrooms": 1,
    "bathrooms": 1,
    "sqft": 650,
    "beds_min": 1,
    "beds_max": null,
    "building_name": "The Modern",
    "total_units": 120,
    "year_built": 2018,
    "stories": 8,
    "amenities": ["In-Unit Laundry", "Dishwasher", "Balcony", "Gym"],
    "pet_policy": "Cats and dogs allowed. No aggressive breeds. Max 2 pets.",
    "pets_allowed": true,
    "cat_allowed": true,
    "dog_allowed": true,
    "dog_size_limit": "50 lbs",
    "cat_deposit": 500,
    "dog_deposit": 750,
    "photos": ["https://...jpg", "https://...jpg"],
    "photo_count": 8,
    "viewed_count": 234,
    "favorites_count": 45,
    "inquiries_count": 12,
    "schools": [
        {
            "name": "Bryant Elementary",
            "level": "Elementary",
            "rating": 7,
            "grades": "K-5",
            "distance_miles": 0.3
        }
    ],
    "listing_type": "Apartment",
    "available_date": "2026-09-01",
    "scraped_at": "2026-08-11T00:00:00.000Z"
}
```

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

***

### 💰 How much does it cost?

HotPads charges nothing to browse, but this Actor uses Apify compute units (CU):

- **Fast mode** (search pages only): ~1 CU per 1,000 listings
- **Detail mode** (with detail pages): ~5-10 CU per 1,000 listings (each listing = 1 extra page)
- **Proxy:** Residential proxies cost extra on Apify (included in free tier limits)

**Tip:** Start with `maxPages: 5` and `includeDetails: false` to test, then scale up.

***

### ⚡ Performance Tips

- **Reduce maxPages** to limit scope. HotPads hard-caps at 40 pages anyway.
- **Use fast mode** (includeDetails: false) for pricing trends — search pages have price, beds, baths, and address.
- **Use detail mode** (includeDetails: true) only when you need full descriptions, pet policies, or photos.
- **Residential proxies** are strongly recommended. HotPads uses PerimeterX bot detection that blocks datacenter IPs.
- **Lower concurrency** (1-2) if you're getting blocked frequently.

***

### 🔧 Architecture

This Actor implements the anti-detect patterns from [tangerine\_loom/hotpads-scraper](https://apify.com/tangerine_loom/hotpads-scraper):

1. **Camoufox launch lifecycle:** Fingerprint options are built inside `preLaunchHooks` (not at module scope), so each browser gets a fingerprint derived from its actual exit IP.
2. **DOM content loaded:** Block detection runs after `domcontentloaded`, not `commit`, so PerimeterX challenge elements actually exist when checked.
3. **State blob extraction:** Data comes from `window.__PRELOADED_STATE__` — more stable than CSS selectors and contains richer data.
4. **Fail loudly:** Missing state scripts throw errors (triggering retries) instead of returning empty datasets.
5. **Session retirement:** `maxErrorScore: 1` retires sessions on first error — once PerimeterX flags an IP, further requests won't succeed.

***

### ⚠️ Disclaimer

This Actor is for educational and research purposes. Web scraping may violate the Terms of Service of the target website. Users are responsible for ensuring their use complies with applicable laws and the target site's Terms of Service. The author is not responsible for any misuse of this tool.

***

### 🐛 Issues & Feedback

Found a bug or have a feature request? Open an issue on the [GitHub repository](https://github.com) or contact support through Apify Console.

***

### 📚 Resources

- [HotPads](https://hotpads.com) — Target website
- [Camoufox](https://camoufox.com) — Anti-detect Firefox fork
- [Crawlee](https://crawlee.dev) — Web scraping framework
- [Apify SDK](https://docs.apify.com/sdk/js) — Actor runtime
- [Article: Your Actor's input schema is its agent tool definition](https://apify.com/tangerine_loom/hotpads-scraper) — Architecture inspiration

# Actor input Schema

## `startUrls` (type: `array`):

HotPads search result URLs to scrape. Each URL should be a search page from hotpads.com. You can add multiple searches (different cities, filters, price ranges). HotPads hard-caps at 40 pages per search (~1,600 listings).

## `maxPages` (type: `integer`):

How many pagination pages to follow per start URL. HotPads hard-caps at 40 pages (~1,600 listings per search). Reduce this to limit the run scope and compute cost. Each page contains ~40 listings.

## `includeDetails` (type: `boolean`):

When enabled, the scraper visits each listing's detail page to extract the full description, photos, pet policies, and all unit data. Disable for faster runs with fewer compute units — search-page data includes price, bedrooms, address, and basic amenities.

## `maxConcurrency` (type: `integer`):

How many pages to process simultaneously. Higher = faster but more likely to trigger bot detection. 2-3 is safe; 5+ risks blocks. Residential proxies recommended above 3.

## `proxyConfiguration` (type: `object`):

Proxy settings for anti-bot protection. Defaults to Residential Proxies. HotPads uses PerimeterX — residential IPs are strongly recommended for reliable scraping.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://hotpads.com/san-francisco-ca/apartments-for-rent"
    }
  ],
  "maxPages": 40,
  "includeDetails": false,
  "maxConcurrency": 2,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://hotpads.com/san-francisco-ca/apartments-for-rent"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("x.com/hotpads-listings-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://hotpads.com/san-francisco-ca/apartments-for-rent" }] }

# Run the Actor and wait for it to finish
run = client.actor("x.com/hotpads-listings-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://hotpads.com/san-francisco-ca/apartments-for-rent"
    }
  ]
}' |
apify call x.com/hotpads-listings-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,x.com/hotpads-listings-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/aws8g1sA5k4TLNLn2/builds/OYYaWp1z7gR6URjr3/openapi.json