Airbnb Crawler

A web crawler for extracting Airbnb listing data from locations and URLs with support for pagination and parallel processing.

Features

• Location-based scraping: Search by city/location (e.g., "Paris--France")
• URL-based scraping: Scrape from specific Airbnb search URLs with filters preserved
• Smart pagination: Automatically clicks "Next" button to traverse multiple result pages
• Parallel processing: Concurrent API requests for faster data extraction
• Comprehensive data extraction: Host details, reviews, amenities, photos, location data, and more
• Error handling: Robust retry mechanisms and graceful failure handling

Input Parameters

Required

• locations: Array of location strings (default: ["Paris--France"])
  • Format: "City--Country" or "City--State--Country"
  • Examples: ["London--UK", "New-York--NY--United-States"]

Optional

• urls: Array of Airbnb search URLs

  • Preserves all filters and search parameters
  • Supports pagination through multiple result pages

• pages: Number of pages to scrape per location/URL (default: 2, max: 20)

• concurrency: Number of parallel API requests (default: 3, max: 6)

• delay: Delay between requests in milliseconds (default: 300ms)

• maxRetries: Maximum retry attempts for failed requests (default: 3)

Output Data

Each listing contains:

Basic Information

• id: Unique listing identifier
• title: Property title
• url: Direct link to listing
• personCapacity: Maximum number of guests
• canonicalUrl: SEO-friendly URL
• iphoneLink: Deep link for mobile app

Host Details

• hostDetails: Host information including name, profile image, superhost status
• Host highlights, response rate, and verification status

Property Details

• overview: Property type and key features
• highlights: Special amenities and standout features
• description: Full property description
• amenities: Complete list of available amenities
• photos: Property images with metadata

Reviews & Ratings

• reviews: Review count and average rating
• ratings: Detailed category ratings (cleanliness, communication, etc.)
• brandDescription: Additional review context

Location & Policies

• location: Coordinates and area information
• breadcrumbs: Location hierarchy
• houseRules: Property rules and restrictions
• cancellationPolicy: Cancellation terms and conditions

Metadata

• scrapedAt: Timestamp of data extraction

Usage Examples

Basic Location Scraping

{
  "locations": ["Paris--France"],
  "pages": 3
}

URL with Filters

{
  "urls": ["https://www.airbnb.com/s/Tokyo/homes?price_min=50&price_max=200&room_types%5B%5D=Entire%20home%2Fapt"],
  "pages": 5
}

Multiple Inputs

{
  "locations": ["Barcelona--Spain", "Rome--Italy"],
  "urls": ["https://www.airbnb.com/s/London/homes?guests=4"],
  "pages": 2,
  "concurrency": 5
}

High Performance Settings

{
  "locations": ["Miami--FL--United-States"],
  "pages": 10,
  "concurrency": 6,
  "delay": 200,
  "maxRetries": 5
}

Performance Considerations

• Rate Limiting: Built-in delays prevent overwhelming Airbnb's servers
• Parallel Processing: Configurable concurrency for optimal speed vs. stability
• Smart Pagination: Automatic detection when no more results are available
• Memory Efficient: Streams data to output without storing large datasets in memory

Error Handling

• Network Issues: Automatic retries with exponential backoff
• Missing Data: Graceful handling of incomplete listings
• Pagination Limits: Stops when no more pages are available
• Invalid URLs: Clear error messages for malformed inputs

Output Format

Results are streamed as individual JSON objects, one per listing. Summary information includes:

• Input parameters used
• Total processing time
• Success/error counts

Limitations

• Respects Airbnb's robots.txt and rate limits
• Some data may be unavailable for certain listings
• Performance depends on network conditions and Airbnb's response times
• Maximum 20 pages per input to prevent excessive requests