🏠 HotPads Rental Listings Scraper

HotPads Rental Listings Scraper is a production-grade Apify Actor that extracts structured rental listing data from HotPads using an anti-detect Camoufox browser with geo-matched fingerprints. It reads data directly from HotPads' server-side state blobs (window.__PRELOADED_STATE__) instead of fragile CSS selectors, providing more stable and richer data extraction.

Perfect for rental market analysts, real estate investors, property managers, and AI agents that need reliable access to rental listing data without manual browsing.

✨ Key Features

• 🦊 Anti-Detect Camoufox Browser: Uses Camoufox (stealthy Firefox fork) with geo-matched fingerprints built inside the browser launch lifecycle — not at module scope. Each browser session gets a fingerprint derived from its actual exit IP, defeating PerimeterX bot detection.
• 📊 Server-Side State Extraction: Reads directly from window.__PRELOADED_STATE__ instead of scraping the DOM. This is more stable across site redesigns and contains richer data (price drops, popularity counters, school ratings) that never renders on screen.
• 🔍 Smart Block Detection: Detects PerimeterX challenges, 403/429 errors, and soft-blocks (missing state scripts). Blocks fail loudly with retries instead of silently returning empty datasets.
• 🐾 Pet Policy Data: Extracts detailed pet policies including cat/dog allowances, size limits, and deposits — data unique to HotPads' state blob.
• 🏫 School Information: Pulls nearby school data with ratings, grade ranges, and distances from the state blob.
• 📈 Price Drop Tracking: Captures price drop amounts and flags from the state blob — data the page never renders.
• 🔧 AI Agent Optimized: Input schema descriptions encode site limits and cost tradeoffs so AI agents can plan runs correctly without human supervision.

🎯 Who is this for?

• Rental Market Analysts: Track pricing trends, availability, and pet policies across neighborhoods.
• Real Estate Investors: Identify price drops and investment opportunities.
• Property Managers: Monitor competitor listings and pricing.
• AI Agents & Automation: Reliable, structured data extraction for autonomous workflows.
• Data Scientists: Clean, JSON-formatted rental data for analysis and modeling.

---

🚀 How to use

Step 1: Get your HotPads search URL

1. Go to hotpads.com
2. Search for rentals (city, neighborhood, filters)
3. Copy the URL from your browser address bar
   • Example: https://hotpads.com/san-francisco-ca/apartments-for-rent?beds=1&maxPrice=3000

Step 2: Configure the Actor

Paste the URL into the Start URLs field. Adjust settings:

• Max Pages: How many pagination pages to follow (each page = ~40 listings). HotPads hard-caps at 40 pages (~1,600 listings per search).
• Include Details: Enable to visit each listing's detail page for full descriptions and pet policies (slower, more compute units).
• Max Concurrency: 2-3 is safe; higher risks blocks even with residential proxies.

Step 3: Run and download

Click Start and wait for the run to finish. Download your dataset in JSON, CSV, HTML, or Excel format from the Output tab.

---

📥 Input

Field	Type	Default	Description
Start URLs	array	hotpads.com/san-francisco-ca/apartments-for-rent	HotPads search result URLs. Each URL should be a search page from hotpads.com.
Max Pages	integer	40	Pagination pages per URL. HotPads hard-caps at 40 (~1,600 listings).
Include Details	boolean	false	Visit detail pages for full descriptions and pet policies (slower, more CU).
Max Concurrency	integer	2	Simultaneous pages. 2-3 is safe; 5+ risks blocks.
Proxy Configuration	object	Residential Proxies	Anti-bot proxy settings. Residential strongly recommended.

---

📤 Output

Each listing is a structured JSON record with explicit null values for missing data (not omitted keys). This ensures downstream tools can distinguish "Actor doesn't return this field" from "data wasn't available."

{
    "listing_id": "12345678",
    "url": "https://hotpads.com/san-francisco-ca/apartments-for-rent/12345678",
    "source_url": "https://hotpads.com/san-francisco-ca/apartments-for-rent",
    "title": "Modern 1BR in SoMa",
    "property_type": "Apartment",
    "address": "123 Main St, San Francisco, CA 94105",
    "city": "San Francisco",
    "state": "CA",
    "zip_code": "94105",
    "neighborhood": "SoMa",
    "latitude": 37.7749,
    "longitude": -122.4194,
    "price": 2800,
    "price_min": 2800,
    "price_max": null,
    "price_drop": 200,
    "has_price_dropped": true,
    "bedrooms": 1,
    "bathrooms": 1,
    "sqft": 650,
    "beds_min": 1,
    "beds_max": null,
    "building_name": "The Modern",
    "total_units": 120,
    "year_built": 2018,
    "stories": 8,
    "amenities": ["In-Unit Laundry", "Dishwasher", "Balcony", "Gym"],
    "pet_policy": "Cats and dogs allowed. No aggressive breeds. Max 2 pets.",
    "pets_allowed": true,
    "cat_allowed": true,
    "dog_allowed": true,
    "dog_size_limit": "50 lbs",
    "cat_deposit": 500,
    "dog_deposit": 750,
    "photos": ["https://...jpg", "https://...jpg"],
    "photo_count": 8,
    "viewed_count": 234,
    "favorites_count": 45,
    "inquiries_count": 12,
    "schools": [
        {
            "name": "Bryant Elementary",
            "level": "Elementary",
            "rating": 7,
            "grades": "K-5",
            "distance_miles": 0.3
        }
    ],
    "listing_type": "Apartment",
    "available_date": "2026-09-01",
    "scraped_at": "2026-08-11T00:00:00.000Z"
}

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

---

💰 How much does it cost?

HotPads charges nothing to browse, but this Actor uses Apify compute units (CU):

• Fast mode (search pages only): ~1 CU per 1,000 listings
• Detail mode (with detail pages): ~5-10 CU per 1,000 listings (each listing = 1 extra page)
• Proxy: Residential proxies cost extra on Apify (included in free tier limits)

Tip: Start with maxPages: 5 and includeDetails: false to test, then scale up.

---

⚡ Performance Tips

• Reduce maxPages to limit scope. HotPads hard-caps at 40 pages anyway.
• Use fast mode (includeDetails: false) for pricing trends — search pages have price, beds, baths, and address.
• Use detail mode (includeDetails: true) only when you need full descriptions, pet policies, or photos.
• Residential proxies are strongly recommended. HotPads uses PerimeterX bot detection that blocks datacenter IPs.
• Lower concurrency (1-2) if you're getting blocked frequently.

---

🔧 Architecture

This Actor implements the anti-detect patterns from tangerine_loom/hotpads-scraper:

1. Camoufox launch lifecycle: Fingerprint options are built inside preLaunchHooks (not at module scope), so each browser gets a fingerprint derived from its actual exit IP.
2. DOM content loaded: Block detection runs after domcontentloaded, not commit, so PerimeterX challenge elements actually exist when checked.
3. State blob extraction: Data comes from window.__PRELOADED_STATE__ — more stable than CSS selectors and contains richer data.
4. Fail loudly: Missing state scripts throw errors (triggering retries) instead of returning empty datasets.
5. Session retirement: maxErrorScore: 1 retires sessions on first error — once PerimeterX flags an IP, further requests won't succeed.

---

⚠️ Disclaimer

This Actor is for educational and research purposes. Web scraping may violate the Terms of Service of the target website. Users are responsible for ensuring their use complies with applicable laws and the target site's Terms of Service. The author is not responsible for any misuse of this tool.

---

🐛 Issues & Feedback

Found a bug or have a feature request? Open an issue on the GitHub repository or contact support through Apify Console.

---

📚 Resources

• HotPads — Target website
• Camoufox — Anti-detect Firefox fork
• Crawlee — Web scraping framework
• Apify SDK — Actor runtime
• Article: Your Actor's input schema is its agent tool definition — Architecture inspiration