Skyscanner Flight Scraper
Under maintenancePricing
Pay per usage
Scrapes Skyscanner flights via the internal web-unified-search JSON API — no browser needed (pure HTTP over residential proxy). Mirrors the jupri/skyscanner-flight technique.
Origin city or airport (IATA code, e.g. TLV, or a city name).
Destination city or airport (IATA code, e.g. HER, or a city name).
Country you're searching from — gives local deals. ISO country code.
Max number of results to push per search (slice, after sort). Leave empty for all.
Run MANY searches in ONE browser session — PerimeterX is passed once and amortized across all of them. Each item: {"depart":"YYYY-MM-DD","return":"YYYY-MM-DD"} (return optional; origin/target default to origin.0/target.0, or override per item with "origin"/"target"). When set, the single origin.0/depart.0 search is ignored. Records are tagged with search_depart/search_return.
Max seconds to poll each search toward status:complete. Higher = more agents populate per itinerary; lower = faster but fewer agents on some itineraries. Default 120.
Stop polling a search once its result count has not grown for this many seconds. Skyscanner currently freezes these searches (never reports 'complete'), so without this every search runs the full poll budget for nothing. 0 = disabled (poll the whole budget).
Shorter stall window applied only when the result count never grew past the first snapshot — the frozen shape. A search that grew at least once keeps the full poll_stall_s window. 0 = use poll_stall_s for both.
Seconds between polls of a running search. Each poll replays the CREATE request, which starts a NEW search server-side, so polling fast can keep killing a search before it matures. Raise this if searches freeze at 10 results. Default 2.
Send no poll requests of our own — only capture the web-unified-search responses the site's own JS fires. Avoids the 'every poll starts a new search' problem entirely. Falls back to active polling if the site fires only once in 20s.
Abort requests to third-party ad and analytics domains (doubleclick, GTM, hotjar, ...). Nothing on a skyscanner domain and nothing PerimeterX-related is blocked. Cuts page load work; measure the 403 rate before leaving it on.
How many of the trips dates may run at once. MEASURED: leave this at 1. Skyscanner degrades concurrent searches from the same session — same 2 dates took 76s sequentially (1 snapshot each, 15 records each) versus 165s in parallel tabs, where every poll took 20-35s and searches stall-stopped at 10 results. 3 was worse still and lost a whole date. Raise only with fresh measurements.
Stop polling once the cheapest limit itineraries have stopped changing for a few consecutive snapshots, instead of waiting for the full result set to complete. Much faster; small risk of missing a cheaper flight that surfaces late. Best combined with a set limit.
Launch a browser on this many residential IPs in parallel and take the first that passes PerimeterX; the rest are closed. 2 hides the ~20s otherwise wasted when the first IP hits an interstitial. 1 disables the race. Higher = more RAM (use 4GB if >2).
Cabin class.
Include nearby origin airports.
Include nearby destination airports.
Proxy networking. Defaults to Apify Residential proxy (required to pass PerimeterX). Sticky sessions are used internally so the cookie mint and every fire exit the same IP.
{ "useApifyProxy": true, "apifyProxyGroups": [ "BUYPROXIES94952" ]}Additional HTTP headers as {name, value} objects, merged into every request.
Reuse a previously-minted PerimeterX session to SKIP the browser. Set this to the session_id from a prior run's KV 'px_session', and pass that run's cookies in dev_custom_cookies. Phase 1 (pure HTTP) then exits the same residential IP the _px3 is bound to. Works only while the _px3 + sticky IP are still alive (minutes). On failure it falls back to the browser.
Restrict pushed output to these field names (in order). Empty = full record.
Named dataset to push to. Supports {ACTOR}, {DATE}, {TIME} tokens. Empty = default dataset.
Wipe the target dataset before pushing.
Keep null/empty fields in output instead of stripping them.
One-off: navigate the /config/ page for the first itinerary and log every XHR that carries pricing_options, so the details endpoint can be captured. Saves the raw hit to KV key 'config_capture'.
LEGACY / optional. Every run now already returns the full agent list inline in each record's agents (via poll-to-complete + snapshot merge), so this is only needed for the extra jupri-shaped pricing_options with agent names/ratings from the /config details endpoint. Costs one browser session + one page visit per itinerary — leave OFF unless you need the richer per-agent metadata.
How many top itineraries (after sort) to enrich with pricing_options when 'Fetch per-agent pricing options' is on.
With 'Fetch per-agent pricing options' on: output ONLY the top-N enriched itineraries and finish immediately, skipping all other itineraries. Fastest/cheapest when you just want the top results in full.
For multi-date (trips) runs, derive each later date's search by swapping the dates in the CREATE request captured on the first date, issued from the same PerimeterX-warm tab. Measured: the navigation itself is 1-2s but waiting for the site's own JS to fire its search costs 16-26s per extra date. Falls back to a normal navigation whenever the route differs or the replay does not return results. Only affects trips runs.
MEASURED CORRECTION: a PerimeterX block does NOT return 403 on the results document. The document loads normally and the site JS simply never fires a search, so both raced IPs sit out the full 30s deadline (~36s per failed attempt, up to 3 attempts per run). The actor therefore probes the PAGE for the PX challenge (captcha element / challenge copy) while waiting, and always LOGS when it sees one and whether that page went on to fire anyway. Set this to how many extra seconds to allow after the signature has been seen twice. 0 = off (keep the full wait) and is the default, because a page mid-challenge can still self-solve. Set it from the "after PX signature: fired=... at Xs" log lines rather than by guessing.
EXPERIMENTAL, off by default. For trips runs, ask for every remaining date's search concurrently from the first date's PerimeterX-warm tab in a single in-page call, instead of one tab + navigation + blocking replay per date. Each date otherwise costs 12-27s of pure server latency and they add up; this makes the total roughly the slowest single call. Unlike search_concurrency (measured worse) it opens no extra tabs and runs no competing site JS. Any date whose replay does not come back usable falls back to a normal navigation.
When a search does not reach "complete", its flights and prices are correct but each itinerary carries only ONE agent, because the inline agent list fills in over poll snapshots that never arrive. /itinerary/details does not depend on the search completing — measured on a frozen date it returned 18-19 priced agents with names, ratings and booking links for itineraries that had 1. This enriches the cheapest N of any PARTIAL date so the rows people actually read keep a full seller list. Costs a few seconds, and only on dates that came back partial. 0 = off. Dates that complete normally are unaffected; fetch_details still overrides for all dates. Sizing note: the enrichment replays fire 4 per round, so 4 costs the same as 3 (one round) and 5-8 all cost one extra round. 4 is the free maximum.