Google Hotels Scraper With Rate Comparison
Pricing
from $5.99 / 1,000 results
Google Hotels Scraper: Extract hotel names, prices, ratings, reviews, locations, amenities, availability, and booking links. Compare room rates across properties and dates to analyze pricing trends, identify competitive offers, and support hotel market research and travel planning.
src/main.py: the 0.6 verification run showed the substring-blocklist fix in is_valid_address()
just made the same ADDR_PATTERNS fallback loop move on to its next candidate match, this time
a fragment of Google's own obfuscated inline JS (e.g. "dvoid 0,R=q(Q,function(U,f){for(f=25;...")
for the same alternative-listing pages (these pages appear to carry no real structured address at
all). Replaced the substring blocklist with a general CODE_LIKE_RE guard (function(, \uXXXX
escapes, for(, var , window./document., etc.) in is_valid_address() so any code-shaped
fallback match is rejected outright (falls back to no address, never a fabricated/garbage one)
instead of chasing individual bad matches one at a time.src/main.py is_valid_address(): the full default-input verification run for 0.5
(1b0eJQjHMLLcP1hJ4, 10/10 hotels saved) showed 2 of the 10 items -- both alternative-
accommodation listings ("Rooftop Penthouse with Terrace, Prime Location", "MOOo by the Castle",
as opposed to traditional hotels) -- got a garbage address value: a <meta viewport> content
string (width=device-width,initial-scale=1.0,...) or an internal Google click-tracking token
blob (...;ved:2ahUKEwj...;track...) picked up by the regex-fallback address matcher when no real
structured address was present on that listing type's page. Added explicit rejection of both
substrings' distinctive markers in is_valid_address()._fill_prices_and_website()'s provider-name/price
extraction is tuned against traditional hotel entity-page markup and does not fully cover this
alternative-listing page variant. This does not affect traditional hotel results (the actor's
primary target and the vast majority of /travel/hotels/entity/... results for a normal
city/hotel-name search) and is disclosed here rather than silently left unfixed.After the 0.3 streaming-push fix, a live re-run with the default 10-hotel input still produced 0
dataset items (run Yr7QNjjvnRXnirW9Y), this time hitting the new per-batch parsing timeout on
every single batch. Added temporary per-hotel diagnostics (page-size + parse-duration logging) and
ran a smaller, fast diagnostic run (fMWnG92wpgNh6IOGd, maxHotels: 2) to isolate the cause.
Findings:
main,
/location, and /details pages -- roughly 8.5MB of real HTML per hotel (confirmed by fetching
the same live URLs directly, outside the actor, which returned real, correctly-structured pages
with the same size in ~1.3s each).extract_hotel_data()'s regex/BeautifulSoup extraction is functionally correct -- run locally
against the same real fetched pages it completed in ~5.6s and returned fully correct data (real
title, 16-34 OTA offers with real prices/deep-links). The bug is not in the parsing logic itself.defaultMemoryMbytes: 1024, Apify's CPU allocation scales with
memory), the identical parse of the identical kind of page took ~79 seconds per hotel --
roughly 14x slower than a normal machine -- confirmed via the diagnostic run's own per-hotel
timing log. extract_hotel_data() runs a few dozen regex passes (address/phone/rating patterns,
find_in_json recursively walking every AF_initDataCallback JSON blob per field) across
~8.5MB of HTML per hotel; on a CPU-starved container this genuinely takes over a minute per
hotel, and with maxHotels: 10 (3 sequential batches of 4/4/2) the cumulative CPU time for
parsing alone reliably exceeds even the widened 0.3 timeouts.So the full, layered picture for this actor's "no output data" symptom is: (1) proxy/URL-pattern bugs from a prior fix (see 0.2 below), (2) the 0.3 fix for the all-or-nothing end-of-run dataset push, and (3) this memory/CPU-allocation bottleneck -- all three needed fixing for the actor to reliably produce a non-empty dataset from a fresh default Console "Start" click.
.actor/actor.json: defaultMemoryMbytes raised from 1024 to 4096. Apify allocates CPU
proportional to memory, so this directly targets the measured CPU-bound parsing bottleneck
(expected to bring ~79s/hotel down toward the ~5-20s/hotel range observed off-container) instead
of just working around it with ever-larger timeouts.src/main.py: added lightweight per-hotel diagnostic logging (fetched page sizes; parse duration
src/main.py: crawl_hotels() parse-stage and fetch-stage per-batch timeouts widened further
(parse: 90 + 90s/hotel; fetch retained from 0.3 at 90 + 110s/hotel) and the outer per-query
timeout in main() kept generous (120 + maxResults*150) to give the (now faster, but still
non-trivial) real-world parsing enough headroom to complete without relying solely on the 0.3
streaming-push safety net.fMWnG92wpgNh6IOGd (maxHotels: 2, still on the 1024MB tier): 2/2 real hotels
saved -- "The Julius Prague" (34 OTA offers, bestDeal Super.com), "Hermitage Hotel Prague" (30
OTA offers, bestDeal Super.com) -- confirming the extraction logic itself is correct once given
enough time; the earlier empty-dataset runs were a pure CPU/time-budget problem, not a broken
parser or a Google markup change requiring new selectors.maxHotels: 10, 4096MB) verification
run ID, item count and sample items.Live-verified via apify call-equivalent run BUolChvq6xcNNrMxa against the unmodified schema-
default input (destinations: ["Prague hotels"], maxHotels: 10, no proxy prefill): the run
finished with platform status SUCCEEDED and exit_code: 0, but the dataset had 0 items. The
log showed 8 of 10 hotels start processing (batches of 4), the first batch completing in a
realistic 169s, then the run terminating with ❌ Timeout processing query: Prague hotels /
💾 0 hotels saved to dataset roughly 590s after start — a classic "SUCCEEDED but actually failed"
case the required methodology exists to catch.
Root cause: crawl_hotels() only appended parsed hotels to an in-memory results list per batch
and pushed everything to the Apify dataset in one loop placed after the entire per-batch loop
finished (i.e. after all batches for the query). That whole coroutine is wrapped in
asyncio.wait_for(crawl_hotels(...), timeout=query_timeout) in main(). Real Google
Hotels/Travel fetch+parse timing (one 4-hotel batch alone legitimately took 169s; the next batch
was still in-flight past 400s once proxy escalation was needed for some of its hotels) reliably
exceeds the old query_timeout = 60 + maxResults*50 budget (560s for the schema-default
maxHotels: 10). When asyncio.wait_for times out it cancels the inner coroutine outright —
crawl_hotels() is aborted at whatever await it is suspended on, so the final push loop (being
strictly after all batches) never ran even once, and 100% of already-successfully-scraped hotel
data (including the first batch's results) was silently discarded. This is a structural
guarantee-of-data-loss on any run whose total processing time exceeds its own self-imposed budget
— which the schema-default 10-hotel input reliably does — independent of whether the scraping
itself is working correctly.
A secondary contributing issue: the parsing stage (extract_hotel_data run in a
ThreadPoolExecutor) had no timeout at all, so a slow parse of a large real Google Travel HTML
payload could add unbounded time with no diagnostic; and both the fetch batch's and parsing
stage's failure paths (isinstance(data, Exception): continue) silently swallowed exceptions
with no log line, making failures invisible in the actor log.
src/main.py crawl_hotels(): hotels are now pushed to the dataset immediately after each
batch is parsed, not buffered until the whole query finishes. A later batch timing out (or the
outer per-query timeout eventually firing) can now only cost that batch's still-unprocessed
hotels — every hotel already fetched and parsed successfully is durably saved first. Non-
relevance sort orders are applied as a best-effort local sort of each batch as it is flushed,
since a true global sort across the whole result set would require buffering everything until
the end again (the exact trap being fixed).src/main.py crawl_hotels(): the batch fetch timeout (asyncio.wait_for around
asyncio.gather(*tasks, ...)) was widened from a flat 220s to 90 + 110s/hotel, and a new
parsing-stage timeout (60 + 20s/hotel) was added around the previously-unbounded
ThreadPoolExecutor gather — both sized against the real batch-1/batch-2 timing observed live.src/main.py crawl_hotels(): exceptions during per-hotel parsing and hotels dropped for
missing a usable title are now logged with Actor.log.warning(...) instead of being silently
skipped, so a genuine extraction failure is visible in the actor log instead of just showing up
as a smaller-than-expected dataset.src/main.py main(): the outer per-query timeout was widened from 60 + maxResults*50 to
120 + maxResults*150, matching the batch timeouts above and giving a normal run enough budget
to actually finish end-to-end (still well under the platform's own 3600s hard run timeout for the
default maxHotels: 10), rather than relying solely on the streaming-push safety net above..actor/actor.json: version bumped 0.2 -> 0.3.destinations: ["Prague hotels"], maxHotels: 10, proxyConfiguration: {"useApifyProxy": false}The actor was returning empty (or near-empty) datasets in production. A sibling actor targeting
the exact same Google Hotels/Travel surface (Google Hotels — Location Details Scraper) had just
been live-tested and fixed for this same class of bug; each of its findings was independently
verified against this actor's own code and target queries and confirmed to apply here too. Three
compounding bugs in the proxy-fallback layer (ProxyManager / make_request_with_proxy_fallback
in src/main.py):
/travel/* — no server-rendered <title>, no AF_initDataCallback/
google.travelData JSON blob, i.e. none of the data this actor's regex/BeautifulSoup parsing
depends on. The old code treated any HTTP 200 as a success, so it would happily accept that
empty shell, extract zero hotel links / zero fields, and never escalate the proxy tier at all.make_request_with_proxy_fallback used one
shared attempt counter (max_retries=3) for both retries and tier escalations. Calling
escalate_proxy() consumed one of those three attempts on its own, so by the time a call
finally escalated up to the residential tier, there was no attempt budget left in the loop to
actually issue a request through it — the function logged "All request attempts failed" and
returned None immediately after successfully obtaining a working residential proxy URL it
then never used.GOOGLE_SERP proxy group.
Live-tested 2026-08-30 against this actor's actual target paths: GOOGLE_SERP refuses the HTTPS
CONNECT tunnel outright (CONNECT tunnel failed, response 400, proxy header
x-apify-proxy-error: true), and even over plain HTTP it returns HTTP 400 "Google query failed: Unsupported Google service or resource"/travel/* path — the group only
serves google.com/search. So the "datacenter" tier was guaranteed to fail on every run,
wasting the one retry budget bug #2 left it before ever reaching residential.proxyConfiguration prefills
{"useApifyProxy": false} (a fast/free default first attempt). ProxyManager.get_proxy_url
merged that straight into every escalated tier's settings too, and the Apify SDK honors an
explicit useApifyProxy: false over any apifyProxyGroups passed alongside it — so for any run
left on its default input, every escalation tier silently produced no proxy at all.Only a RESIDENTIAL-tier Apify proxy was confirmed (via direct live requests, including through
this account's own proxy credentials, and via the sibling actor's identical live testing against
the same surface) to reliably return the real, server-rendered page for google.com/travel/search,
.../entity/<id>, .../entity/<id>/location, and .../entity/<id>/details — direct and
default-datacenter-pool requests intermittently receive the empty client-render-only shell
described above.
prices
array — its entire differentiator — still came back empty. Live-tested 2026-08-30 against a real
google.com/travel/hotels/entity/<id> page (Hotel NH Collection Prague Carlo IV, Prague): the
page's OTA offer links now use the href path /travel/lodging/clk?..., not the /travel/clk?...
path this actor's _fill_prices_and_website() matched (href=re.compile(r'travel/clk')).
'/travel/lodging/clk...' does not contain 'travel/clk' as a contiguous substring, so the
pattern silently matched zero <a> tags on every run — every OTA offer, bestDeal,
priceSpreadAbs/priceSpreadPct, and taxes/fees were dropped regardless of which proxy tier
successfully fetched the page. The two 'travel/clk' not in href exclusion checks in
_fill_website_fallback() (meant to avoid mistaking a booking deep link for the hotel's own
website) had the same stale-substring problem, though it was benign there since it fails safe.src/main.py: added _is_blocked_page() — a content-based check (missing <title>, missing
AF_initDataCallback/google.travelData, the EU consent interstitial, or a too-short body) used
alongside the HTTP status code to decide whether a response actually needs a proxy escalation,
instead of trusting a 200 status code alone.src/main.py: rewrote make_request_with_proxy_fallback() so each proxy tier gets its own small
retry budget (max_retries_per_tier=2) and escalation only happens once that budget is exhausted
— an escalation now always leaves at least one real request against the newly obtained proxy,
instead of consuming the attempt that would have used it.src/main.py: ProxyManager — removed the GOOGLE_SERP tier (confirmed unusable for this
surface, see above); the "datacenter" tier now requests Apify's default shared proxy pool (no
apifyProxyGroups, since there is no explicit "DATACENTER" group name); the escalation ladder
is now none -> datacenter -> residential. Also added an asyncio.Lock around
escalate_proxy() since several concurrent hotel fetches (main + /location + /details,
several hotels per batch) share one ProxyManager instance and could otherwise race each other
through more than one tier jump at once.src/main.py: ProxyManager.get_proxy_url() now always forces useApifyProxy: True when
building an escalated tier's settings, regardless of the run's own proxyConfiguration input —
fixing the default-input footgun described above so escalation actually takes effect on an
unmodified default run.src/main.py: crawl_hotels() now sends a CONSENT cookie on every request, bypassing the EU
"Before you continue to Google" consent interstitial that otherwise returns a titleless page and
makes extraction fail from EU-region proxy IPs.src/main.py: added TRAVEL_CLK_HREF_RE (travel/(?:lodging/)?clk) and used it in
_fill_prices_and_website() and both _fill_website_fallback() guards, replacing the stale
'travel/clk' literal so OTA offer links under Google's current /travel/lodging/clk path are
found again (old /travel/clk markup, if Google ever serves it, still matches too).README.md: corrected the anti-bot-handling FAQ answer, which claimed escalation only happens on
HTTP 403/429/503 — it now also documents the content-based (200-status empty-shell) trigger.crawl_hotels() against google.com/travel/search?q=Prague+hotels and the
resulting hotel entity//location//details pages with the rewritten proxy-fallback logic.apify call run against production (destination "Prague hotels", 1 hotel, real
arrivalDate/departureDate) returned real, complete data end to end after both fixes: hotel
title "Hotel NH Collection Prague Carlo IV", real address, phone, gps, aboutHotel, and
— the actor's differentiator — a 20-entry prices array (providers including NH Collection
Prague Carlo IV/official site, Agoda, Vio.com, Expedia.com, Slevomat.cz and 15 more), a non-null
bestDeal (Agoda), numberOfOffers: 20, priceSpreadAbs/priceSpreadPct both populated, and
a real taxesAndFees object read straight from Agoda's own deep-link (tax: 3026.81).
Before the TRAVEL_CLK_HREF_RE fix, the identical live page produced prices: [],
numberOfOffers: 0, bestDeal: null — confirming that fix, not just the proxy fixes, was
required to restore the actor's core output.Initial release.