The FlipKey scraper

Hello! I'm Indy, and will be your guide in this actor description. They said I had to be the most descriptive as possible and I know it can become annoying after a while, but the good news is that you don't need to read this at all because the interface is pretty intuitive and modifying things you're not fully aware of cannot break anything, so chill. The actor we're reviewing grabs listings from FlipKey, which is a popular online vacation rentals marketplace, funded in 2007 and now owned by TripAdvisor, a HUGE travel platform itself. I hope that the following information is a good use for you and that you quickly find what you're looking for. Happy scraping! 🙂

Let's talk about the Input

The only thing you need to enter before running this actor is the Location. Most of the time is just matter of writing a place name and you'll be fine. However, sometimes is better to be a little more descriptive about what you want.

As a general rule, if you write a simple keyword, the actor will choose the first relevant suggestion that is shown by flipkey.com, but sometimes this may not be the one you need and because of that, going to the website and seeing the suggestions list yourself is a big help on choosing the right keyword, avoiding you to waste time looking for listings in probably unrelated locations.

To show you a real example, if you are looking for listings in Rome, and you type rome in the location, the actor will pick Province of Rome, Lazio, Italy, which I'm certain it's exactly what you had in mind.

However, let's say you are now interested in Newcastle, but the one in Australia. If you go to flipkey.com, and in the Where do you want to go? text box, you type newcastle, most of the suggestions are in UK.

In this case it's better to type newcastle, australia in the actor's location to help it to accurately pick the most suitable target.

Finally, have into account that to shorten the list of possible matches, you may also specify a region or state before (or instead of) a country name. E.g., if you are looking for Bristol in Tennessee, bristol, tennessee is enough to get you the right location and does not require the , united states part.

Filtering your listings

The check-in and check-out dates are the first things you should specify to shorten your search hits, and are specifically required if you need to scrape booking quotes.

You may pick dates as nearer as tomorrow itself, but as you may expect, these kind of last-minute bookings (if found) can be gone quickly or need of some special procedure by the host.

Entering the number of guests is also mandatory for booking quotes, obviously. But for most cases, the default values will be more than enough.

If you want exact quotes, be as detailed as possible.

The required number of bedrooms and bathrooms can be left empty, but just like with guests, selecting what you're looking for may reduce the number of unwanted listings.

In terms of specifying those numbers, you always select a minimum required. I.e., it's possible to get listings with more rooms than what you actually need.

Get the most out of your budget by filling out this value.

Just as its name says, this is the top final amount intended to be paid for every booking, without the refundable deposits, which also appears in the output as a separated value, if you ask.

As we'll see soon, even if you don't instruct the actor to scrape booking quotas, you may get a 'preliminary' total, which is just the number of nights (according to the check-in/check-out dates) multiplied by the listing's price per night (PPN), with fees and deposits left out of the calculations.

Booking quotes can take additional actor run-time (not that much!) and maybe for some people these basic values are more than enough.

Deciding what information to collect

These three options will populate the basic listing with additional details or just empty/default values when the respective group is disabled.

• Additional information will fill these details when set:

  • Property description (basic HTML content, as shown in the respective web page)
  • Minimum stay (a short line, as specified by the host)
  • A summary of the available rooms in the listed property
  • Features that make (or make not) this property ideal for families
  • Features that make (or make not) this property accessible by car, foot, etc.
  • An overview of the property's guest ratings (from 'excellent' to 'terrible')
  • Host name, as it appears in the listing
  • Languages spoken by the host
  • Check-in / check-out times, in 24-hour format
  • Payment methods & terms
  • Is smoking allowed?

• Booking quote will fill these details when set:

  • Stay length (number of nights)
  • PPN
  • Website fees
  • Host/owner fees
  • Damage deposit (refundable and not included in total)
  • Reservation deposit (payment needed to secure the booking)
  • Total

• Nearby places:

  • In its simplified/optimized form, this fills a composite element consisting of:
    • Place name
    • Category ('restaurant' or 'attaction')
    • Address
  • In its non-optimized form, a more complex structure will be gathered. For example:

    1"nearby": [
    2    {
    3        "name": "Sintesi Ristorante",
    4        "category": "restaurant",
    5        "subCategories": [
    6            "sit_down"
    7        ],
    8        "types": [
    9            "italian",
    10            "contemporary",
    11            "healthy",
    12            "seafood",
    13            "international",
    14            "mediterranean"
    15        ],
    16        "address": "Viale Dei Castani 17, 00072 Ariccia Italy",
    17        "distance": "5.89",
    18        "rating": "5.0",
    19        "ranking": "#1 of 89 Places to Eat in Ariccia",
    20        "priceLevel": "$$ - $$$"
    21    },
    22    {
    23        "name": "Minardi Frascati Winery",
    24        "category": "attraction",
    25        "subCategories": [
    26            "food_drink",
    27            "activities"
    28        ],
    29        "types": [
    30            "wineries"
    31        ],
    32        "groups": [
    33            "Food & Drink"
    34        ],
    35        "address": "Via San Marco 2a, 00044, Frascati Italy",
    36        "distance": "3.66",
    37        "rating": "5.0",
    38        "ranking": "#1 of 20 things to do in Frascati"
    39    }
    40]
    

    For simplicity, only two places were included in the previous sample

Yet even more filters

If you feel a little picky, there are 4 main feature groups you may use to tell your desired scraped properties apart (the list of selectable options is considerably big):

• Rental types (anything imaginable is here)

• Amenities (from essentials like WiFi to extremely essentials like a tennis court)

• Suitabilities (a set of accessibility and family-friendly features)

• Activities (fun stuff to do on-site and around)

You are free to choose all that apply

That obscure advanced section

Don't panic! Long story short, you may leave everything inside untouched, but if you are tempted to mess with the actor's behavior check the following definitions.

I'm certain this is the option you will end modifying sooner or later. By default, this actor will gather up to 50 property listings, but if you need more (or less) results, you can set it to your desired limit.

Notice that 'limit' is the magic word here. No matter how high your limit is, if your configured filters are specific enough, the actor can finish with just a few collected listings even after going through several pages of results.

On the other side, you may set this maximum to zero as well, but be wary that scraping with no limits can mean an increase of the actor's CPU & memory requirements. More on this later.

To figure out what's all this stuff about, let's start by defining that successively during the actor's run, this one is gonna be REQUESTing information from FlipKey. For example, requesting location suggestions for the place you supplied keywords to look for, requesting a batch of properties that match your search criteria, requesting the booking totals for a given property, etc. Hence, take a 'request' as a single operation this actor is divided into.

During the usual operation of the actor, many requests will be done and some of them will fail due to a wide number of explanations like random network problems, etc. And based on that:

• Request timeout:

  Is the amount of time we give a single request to succeed. After that, we just forget it.

• Request retries:

  Is the number of times we try again before considering a request as something impossible to do.

• Request cooldown:

  Is a short lapse where we reasonably wait before trying again a previously failed request.

Let me tell you a secret. You may disable the proxy usage for this actor, but it's inadvisable since the multiple requests need to go through 'something' the way the scraping operation looks more like different people accessing the target site instead of an automated program crawling around.

Having said that, you have these choices:

• Custom proxies:

  If you bought proxies from some 3rd-party, this is the place to configure that. Enter one or more URLs with the format http://username:password@my-proxy.example.com. If you only have one URL, that's probably because your provider automatically rotates through multiple addresses, which is fine.

• Automatic proxies:

  Let Apify to decide what's better for you. You are free to select a single country to simulate the origin of the actor requests, but no need to bother with that.

• Selected proxies:

  Go for the residential ones if you are willing to set this up yourself. Those have the best chances of getting successful requests.

These two modify the way the actor's output and log look.

• Dataset optimization:

  When enabled, all complex elements like arrays and objects will be condensed as simpler single values. For example, the list of property's amenities is collected as an array. But since each listing can include several amenities, you could end having to deal with ten or more additional 'columns' (one amenity per column) after exporting the resulting dataset. This may become really overwhelming for a visual browse of the data, specially if you think of the other complex elements which also may add a lot of columns themselves.

  Simplifying an array is just matter of placing all its elements 'in a single cell' (each on its own line). As for objects, the process is not that simple since not everything is included in the 'conversion'. Look above for the Nearby places in the listing details for a short explanation and a sample of this.

  This is how the Amenities could look non-optimized:

  amenities/0	amenities/1	amenities/2	amenities/3	amenities/4
  Wireless Internet	Air conditioning	Satellite TV
  Grill	Patio	Deck	Fireplace	Safe
  ...	...	...	...	...

  
  

  And this is how the same amenities would look optimized instead:

  amenities
  Wireless Internet Air conditioning Satellite TV
  Grill Patio Deck Fireplace Safe
  ...

  
  

  Non-optimized datasets are good for a programmatic post-processing of the results

• Show warnings:

  If something horrible is happening with the actor, like it's failing a lot, etc., you may activate this before sending your logs to me, the way I can get some additional information about what's happening over there.

  These are some real samples of the warnings you may see in the log:

  
  

  !! request failed: [AbortError] The operation timed out ... !!

  Low quality proxies use to timeout a lot, but this may also be caused by a temporary network problem with FlipKey. However, if you are still getting results, this warning can be ignored with no consequences. Maybe use a bigger Request timeout and retry.

  
  

  !! request failed: [HTTP429] Too Many Requests ... !!

  If you are using proxies, several warnings like this are probably due to the service not allowing you to make a big amount of requests, and in such case, the first solution is raising the Request cooldown, and the next thing to do is getting a better service.

  If you are not using proxies, go grab some.

  
  

  !! request failed: [HTTP403] Forbidden ... !!

  We have been caught here! The specific proxy used for this request has been banned. Either temporarily or forever. I reckon this message looks scary, but as long as we have enough proxies to choose from, that's not a worry at all.

  
  

  !! request failed: [FetchError] blah blah blah !!

  Rare. A serious (let's hope temporary) problem between the proxy (or Apify) and FlipKey. An isolated log line like this one should be ignored. But if you are getting too many in a row, consider aborting the actor and try again after some minutes.

  
  

  !! The actor is not using proxies !!

  Guess what this means 🤭

The usual Apify options at the bottom

This section may become important if you plan to scrape a ton of listings. You can get tens easily under the 5-min mark, but beyond this, fine tuning is required:

• Timeout:

  Don't mix it up with Request timeout. This is the total time an actor is given to finish, and it's right there to prevent an excessive, probably unexpected, consumption of the computing resources.

  This actor can run with No timeout enabled and it's technically impossible that it gets stuck forever, but if you still want to keep an eye on the platform usage, I'd recommend a 10-min timeout for a few hunderd of results or one hour for a few thousands.

  Notice two things:

  1. This value is an upper limit. The actor can perfectly finish earlier.

  2. The number of listings you gather depends on the filters you set and what properties are available in FlipKey for the location you chose. It's not about how long the actor runs; it's all about the filters and what's available!

• Memory:

  The Apify's default used to be 4 GB, but this actor is kinda cheap to run with less memory.

Phew! In the end! The Output...

All fields

This is the complete list of scraped fields:

Field	Description
id	FlipKey's listing id, for internal use, but useful to tell rows apart
type	house, apartment, room, etc.
url	direct link to the listing
title	listing's title
description	a long listing description, as written by the host
numBedrooms	actual number of bedrooms in the property
numBathrooms	actual number of bathrooms in the property
maxGuests	how many guests - keep your enthusiasm off, as hosting parties is usually a big no-no!
minStay	minimum possible stay length, as stated by the host
mainPhoto	...
photos	...
rooms	detailed room-by-room description
amenities	...
families	...
access	...
location	specific area where this property sets up shop
latitude	pinpoints your destination if you like receiving orders from an automated voice
longitude	pinpoints your destination if you like receiving orders from an automated voice
numReviews	how many reviews this property has got so far
reviews	total reviews, grouped by rating
avgRating	property's average rating. You don't say...
host	host's nickname or whatever they want to be called
answerTime	a little text specifying how long it takes the host to respond - sometimes undisclosed
languages	list of languages spoken by the host. Allegedly...
checkInTime	the suggested check-in hour - arriving late is no problem
checkOutTime	the suggested check-out hour - departing late is THE problem
payment	a mostly fixed text stating how you should pay for this booking
smoking	is the host welcoming those who enjoy an 'occasional' smoke break?
nearby	a complex nearby structure, or just a simple list of places - you decide
isProtected	I have yet to find a property which is not payment-protected, but who knows
checkIn	specified check-in date, at hand, in case you forgot
checkOut	specified check-out date, at hand, in case you forgot
numAdults	specified number of adults traveling, at hand, in case you forgot
numChildren	specified number of children traveling with you, to edit, because you forgot, right?
numNights	your stay length, to avoid you the embarrasment of counting with your fingers
priceNight	price per night, but it varies depending on the season, as you would expect
bookingTax	some taxes you sometimes have to pay for somewhat reason, somehow...
bookingFee	FlipKey's well-earned fee
ownerFees	additional fee the host pockets, because they can
damageDeposit	you didn't host that party, right? So you can have this money back
reserveFor	pay this and the booking is yours!
totalBooking	grand total, not including refundable deposits

Output views

There are four types of Output 'views' you can browse and export your results based on:

• Tiny:

  The most simple layout, with type, rating, booking total and a link for you to go see the rest.

• Narrow:

  Same as Tiny but including title, location, total reviews and PPN.

• Medium:

  Same as Narrow but including room descriptions, amenities, reviews, languages spoken and all the booking fees and amounts.

• Wide:

  Same as Medium but including the property's description, all the photos, max. guests, min. stay, host information and all the booking details.

That's it. Let's rock!