Searching for a business by its exact name returned nothing. "Sheraton
Kampala Hotel", "The Aleph Rooftop Restaurant", "Lombardi's Pizza" — any query
Google matches to ONE business came back empty, and the run ended with "Google
returned no places for …". The places were there all along: when a query has a
single unambiguous match, Google does not send a result list at all — it sends
that one place from a different branch of the response, and the parser only ever
read the list branch. It now reads both.
This affected exact-name lookups only; searches for a category ("pizza",
"hotels in Kampala") were never touched, and their results are unchanged.
- A locality is not a business. When Google cannot match a query at all it
answers with the surrounding city from that same branch — searching nonsense
in Kampala hands back Kampala. Those are dropped, so a failed search still
reports nothing found instead of inventing a place named after the location.
/maps/place/<Name>/… links in startUrls now work. The business name is
right there in the URL, so it becomes a search term instead of being ignored.- Shortened links (
maps.app.goo.gl/…) now work too — resolved to the real
URL, then handled like any other place link.
- Only place links with no name in them (
/maps/place/?q=place_id:…) are still
unsupported, and the warning now says what to copy instead.
- Dropped the "Only 2048 MB of memory" warning. It fired on the default memory
setting and read like a problem when nothing was wrong.
The scraper is now also a live HTTP API. Standby mode keeps a container warm
and answers requests from it, so places come back in the response body instead
of a dataset you have to poll for:
- An interactive Standby tab on the Actor page, generated from a real
OpenAPI 3 specification (
.actor/openapi.json, also served by the API at
/openapi.json): every endpoint and parameter documented, requests sendable
from the browser with authentication already filled in.
- Endpoints:
GET/POST /search, plus / (usage), /health and
/openapi.json. POST takes the Actor's own input object as its body, so a
saved input runs over HTTP unchanged; query parameters override it field by
field. Short names (q, location, limit, contacts, reviews, images,
minStars, onlyWithEmail, …) map onto the input schema, and list parameters
accept both ?q=gym&q=spa and ?q=gym,spa.
- A request always answers. Apify closes a Standby request after five
minutes, so each one carries a time budget (
timeout, default 120s, max
280s); when it runs out the crawl stops and returns what it already found with
truncated: true. A partial answer beats a 504.
- API defaults differ from run defaults on purpose —
limit 20 (capped at
1000) and deepSearch off, because something is holding a socket open. Big
jobs still belong in a normal run.
- Unknown query parameters come back in
warnings instead of failing the
request, and every rejection (no q, a filter combination that can only
return nothing, a malformed number) is a 400 that says what to change.
- Billing is unchanged: the same per-place, per-contact, per-review and
per-image events a normal run charges.
Internally main() split into the entry point and run_scrape(), which both
modes now share — one HTTP request drives the exact same pipeline a run does,
with the finished rows appended to a list instead of pushed to a dataset. Batch
runs are untouched.
Popular times came back empty for every place. The column was in the dataset
on 100% of rows and every one of them was {} — the histogram was never
actually fetched. /maps/preview/place answers a cold session with a stub:
the place array is all there, but index 84, the popular-times slot, is empty. It
looked exactly like "Google has no histogram for this place", so it failed
silently.
- What unlocks the full payload is the session, not the pb. Google promotes a
session only after a
GET /maps — the one response that hands out an NID
cookie; the search endpoint never does — followed by a couple of search pings,
and it has to happen on that session's own exit IP. The consent cookies
the actor copies in at startup don't substitute: that warm-up runs unproxied,
so its jar has no NID in it.
- Each Google session now promotes itself, once, lazily, right before its first
popular-times request. A run without
popularTimes pays nothing for it, and a
session rotated in after a 429 promotes itself on first use.
- Measured on the same input before and after, Kyiv malls + supermarkets:
0/16 places with a histogram → 16/16. Places that genuinely have none
(small appointment businesses, most of them) still return
{} — that part was
never a bug.
- A run with
popularTimes off no longer carries the field at all. It used
to be written as an empty object on every row, so the console showed a
popular_times column filled to 100% with "0 fields" — a switched-off add-on
looking exactly like a broken one. Empty objects are now dropped from records
the same way empty strings and empty lists always were.
A city could be skipped for being one place short. Deep search only drilled
a location when Google's answer filled a whole page — exactly 100 places. Bucha
returns 93 for "барбершоп", so it was skipped entirely; on an earlier run where
Google happened to return 100 for the same query it WAS drilled, and those
sub-areas found 272 more places. Same input, same actor, a third fewer
results, and not a line in the log to say why.
-
The threshold is now 80, chosen by running the same 32-city input at each
setting rather than by feel. Judged on places found, because how many carry
an email — and so survive skipPlacesWithoutEmail — swung 3.9%-5.8% across
identical runs and is a property of the web, not of this setting:
| threshold | found | time | rate-limit hits |
|---|
| 100 (old) | 3749 | 103 s | 6 |
| 80 | 3975 | 180 s | 143 |
| 70 | 3595 | 315 s | 524 |
| 50 | 3055 | 624 s | 986 |
Going lower backfires: the extra sub-areas provoke enough throttling to lose
more places than the drilling finds.
-
Skipping the drill is logged now, with the count and the threshold, instead of
being a silent decision.
The 429 storms. Rate limiting was self-inflicted and self-sustaining. Three
things fed it, all fixed:
- A failed wave walked the crawl deeper. When every page of a wave failed to
arrive, the code advanced the offset — treating a delivery failure as evidence
that the current pages were exhausted. So each throttled moment pushed the
crawl to offset 300, 400, 800… Measured directly: past offset 200 Google
answers
200 OK with zero places for these queries, so all of that was
requests spent on nothing, which drew more throttling. A failed wave now
retries the same offset.
- Each 429 cost five retries. Five attempts per page, three pages per wave,
twenty-four workers — one throttled moment became hundreds of extra requests,
which is the opposite of what a 429 asks for. Throttling now gets two attempts
before the page is given up; real network failures keep the full budget.
- Barren sub-areas were walked to the end. Deep search splits a city into
postal codes and streets, and a regional city yields two dozen of them that
return nothing. After six barren sub-areas in a row the drill for that location
stops; a sub-area that does yield resets the counter, so large cities are still
drilled fully.
Popular times. New popularTimes add-on attaches Google's weekly busyness
histogram to each place — every hour of every day, with the localized
"Usually not too busy" labels, plus the live "Busier than usual" reading when
Google has one. Included in the Place event; there is no separate charge.
It costs one extra request per place, because Google deliberately keeps this out
of the search response: place[84] is present there but always empty, and the
flag that would fill it makes the search endpoint reject the request. The data
comes from /maps/preview/place, keyed by the hex FID the search already
returns. Places without enough foot traffic have no histogram at all and come
back with an empty popular_times.
- 429 handling no longer burns exit IPs. A pagination wave fires several offsets
down one session, so a throttled IP answered 429 several times within
milliseconds and each one rotated the session — 80 rotations a minute to
escape a single hot IP. Rotation is now rate-limited per session; the pacer's
backoff handles the rest.
Fixed the stall. A long run went silent mid-crawl — CPU pinned, no requests,
no log output. The watchdog added in 1.7.1 named it: the event loop was being
held by synchronous work for ten seconds at a stretch.
- The cause was one regex.
EMAIL_RE was run across the whole page, and its
leading character class matches nearly every byte of HTML, so the engine chewed
a long prefix at every offset only to fail on the missing @. Measured: 181 ms
for a single 206 KB page, and a lead-gen run parses dozens of pages back to
back. Addresses are now located by their @ (a C-level str.find) with the
regex only running over a short window around each one, and a local part is
capped at the 64 characters RFC 5321 allows. Same addresses, 176 ms -> 1 ms
per page.
- Page parsing above 250 KB is handed to a worker thread, so even a pathological
page cannot hold the loop.
contacts_test.py covers extraction (mailto, embedded JSON, Cyrillic pages,
minified CSS) and enforces a per-page CPU budget, so this cannot creep back.
Runs at 2 GB were hit hardest — Apify gives that 0.5 of a CPU core. 4 GB is still
the recommended setting.
Dependency versions are pinned. requirements.txt asked for
curl_cffi>=0.7.0 and apify>=4.0.0, so a build only stayed reproducible while
Docker reused the cached pip layer. The first build to rebuild that layer since
16 August jumped curl_cffi 0.16.0 -> 0.16.2 (published the day before) and the
next run stalled mid-crawl: CPU burning, network flat, no further log output.
Both packages are now pinned to the versions this actor is known to complete
runs on.
- A stalled run says so. A watchdog reports "no new place for Ns" with the
crawl's internal state, and separately reports when the event loop itself was
blocked — which distinguishes "Google is slow" from "something is spinning
inside the process". Previously the log just stopped.
Language and country are detected, not configured. Both used to be input
fields defaulting to en / us, and a run that didn't change them was geo-biased
to the United States no matter where it was searching. countryCode is gone from
the input and language now defaults to Auto.
- The country (Google's
gl) is read from the locations you typed —
Бровари, Київська область, Україна searches as Ukraine — and also from a
gl= parameter, a google.com.ua-style domain in startUrls, or the
country field. Locations in several countries produce no country bias at
all, instead of quietly favouring one.
- The language (
hl) follows the script and spelling of what you typed:
Ukrainian input returns Ukrainian categories and reviews, English input
returns English. Pick a language in the input to override it.
- Two-letter tails are no longer mistaken for countries.
Chicago, IL was
searched as Israel and Los Angeles, CA as Canada; US states, Canadian
provinces and UK nations now resolve to US / CA / GB.
- Pasting a JSON array into the Locations or Search terms box works. Every line
arrived as
"Kyiv, Ukraine", — the trailing comma hid the country and the
quotes went into the query.
- Free-plan runs no longer pin the residential proxy to the US by default; the
exit IP follows the resolved country, and stays unpinned when there isn't one.
language: "zh-CN" was rejected by the actor's own validation even though the
input schema offered it. The language list also grew from 14 entries to 45.
An empty result now always explains itself. Stress-testing the actor with the
input a confused user actually sends turned up three settings combinations that
returned zero rows with nothing in the log to say why — the kind of thing that
reads as a broken actor.
- Contradictory settings are caught before the crawl starts: asking for "only
places with an email" with email scraping off now turns scraping on; combining
it with "only places WITHOUT a website" stops the run immediately, because
emails come from websites and that pair can only ever return nothing.
- When filters empty the dataset, the log says which filter did it and how many
places it removed ("Filters removed 300 of 300 places found: 213 fewer than
500000 reviews, 87 rating below 4.5").
- Filters no longer cripple deep search. Sub-areas are now derived from what
Google returned rather than from what survived filtering, so a strict filter
can't silently disable the geo-drill — and a fully-filtered page is no longer
mistaken for a failed crawl and retried three times for nothing.
maxItems: -5 capped the run at a single row; negative values are now treated
as "no limit" with a warning.countryCode: "USA" and language: "english" are corrected to us / en
instead of being silently ignored by Google.- Unusable entries in
startUrls are reported rather than dropped in silence,
and links to a single place say so explicitly (only SEARCH links are supported).
- A run can now start from
startUrls alone — the schema previously demanded a
search term, which made that feature impossible to use.
- The input schema opens with a quick start, the legacy single-word category
filter is gone from the UI, and
skipPlacesWithoutEmail now warns that it
typically removes 50-60% of results.
Multiple locations. locationQueries takes a list, and every search term is
run against every location — 6 terms x 38 cities is 228 searches in one run. The
single-location fields still work and simply produce a one-entry list.
- New
maxCrawledPlacesPerSearch: a cap per (term, location). With many
locations this is what keeps coverage even — without it the searches that
happen to run first can spend the whole maxItems budget and the remaining
cities return nothing. Both caps are enforced together.
- The country is taken from the locations when they all agree and contradict
countryCode. Searching "Gym in Jaisalmer, Rajasthan, India" while geo-biased
to the US is just the wrong query: measured on Rajasthan towns, gl=in
returned ~8% more places. Results are also stamped with the right country
instead of being labelled US.
- Progress is logged per search ("Search 12/228 done"), and the number of
parallel searches scales with memory instead of being fixed at 3.
Fixed: runs that produced no output. With the 1.4 memory scaling the crawler
was allowed 1200 places in flight against 140 website fetchers, so it ran ~1000
places ahead of enrichment, claimed every publish slot, and nothing reached the
dataset — the run looked frozen while it was only queued behind itself. Places
in flight are now tied to fetch concurrency, and every place has a hard 28s
enrichment deadline after which it is published with whatever was collected.
Measured (New York, 1000 places, 8 GB): first rows at 23s, all 1000 at 74s, 46%
with an email — previously zero rows after 90s.
Concurrency now scales with the memory the run was given. Apify allocates CPU
in proportion to memory (4096 MB = 1 core) and this actor is CPU-bound, so the
4 GB tuning was actively slower on a smaller box — at 1024 MB a 300-place run
took 158s, against 38s at 4096 MB. The actor now reads its memory allocation,
re-tunes every concurrency knob, and warns in the log when it has under 4 GB.
Measured effect at 1024 MB: 158s → 67s for the same 300 places.
- Deep search no longer re-parses places it already has. A New York run parsed
~3,300 listings for 1,000 unique places, because overlapping ZIP searches
return the same restaurants; place IDs are now checked before the per-place
parse. A page that is entirely duplicates keeps pagination going instead of
being mistaken for the end of Google's results.
- A website that fails to load no longer triggers three more blind guesses
(
/contact, /about, …) at the same dead host — that turned one dead site
into four timeouts.
- Website fetches no longer retry timeouts. Retrying is right for Google behind
a rotating proxy, but a business homepage that did not answer in time is
simply not answering, and the retry doubled the cost of the slowest sites.
- Each place now has a hard ceiling on total contact-fetch time, so one
uniformly slow host cannot hold an enrichment slot indefinitely.
- Contact-fetch concurrency is capped at 140 even when memory would allow more:
past that the run competes with itself (curl resolves DNS on a thread per
request) and the extra timeouts show up directly as lost emails.
Measured on the platform (New York, restaurant, 1,000 places, deep search):
search alone finishes in ~36s; with website contact scraping the run takes
~100s at 8 GB. Business sites answer far slower from a datacenter than from a
home connection, and that fetch — not Google — is what the time is spent on.
~20x faster. 1,000 places in New York with website contact scraping now takes
~48s (measured end-to-end against the live endpoint), against ~4.5 minutes
for 295 places before. Without contact scraping the same 1,000 places arrive in
under 10s.
- 100 results per search request instead of 20. Google's
/search?tbm=map
honours !7i100, so one request now returns what used to take five. A single
location is exhausted in 2 requests rather than 8-10.
- Deep search splits by postal code, not by street address. The old sub-area
heuristic read Google's address string as
<street>, <city>, ... — but the
string starts with the business name, so it harvested things like
28 Liberty St 60th floor and queued ~100 of them per location, each returning
zero places. Splitting by ZIP returns ~88 new places per request.
- Contact fetching is no longer capped at 10 connections. curl_cffi's
AsyncSession defaults to max_clients=10, which silently overrode the
configured concurrency and was the run's real throughput ceiling.
- Request timeouts are retried. libcurl reports
Operation timed out, which the
retry matcher (looking for timeout) never matched — so the most common
transient failure was treated as fatal. A timed-out root crawl could end a
deep search with a single partial page.
- Deep search is on by default, and the drill pool is sized by available work
(up to 48 workers) rather than by session count. Default concurrency 3 → 10.
- The drill stops as soon as
maxItems is reached instead of walking the rest of
a queue of now-pointless locations.
- Per-location paging no longer ends early when downstream rejections
(
skipPlacesWithoutEmail, duplicate emails) hand budget slots back.
- Sub-area candidates are anchored to the root location — depth-2 was building
10038, 10013, New York, a ZIP inside a ZIP, which resolves to nothing.
- Default memory raised to 4 GB. On Apify the CPU share is proportional to
memory (4 GB = 1 core), and at 512 MB the actor had 1/8 of a core for parsing.
maxItems is now a hard cap in every mode. Deep search used to crawl and
push a whole location before trimming, so a run capped at 100 could return (and
bill) ~198 places. Result slots are now reserved before a place is enriched,
so you are never charged for extras. maxCrawledPlacesPerSearch is gone —
maxItems is the single result limit.
Much faster. A deep search for barbershop in Minsk capped at 100 went from
5m41s (first row only at the very end) to ~18s with the first row in ~8s.
- Streaming pipeline: places are enriched and pushed as they're found instead of
waiting for a whole location to finish crawling and enriching.
- Search pages are fetched in parallel waves; flat search now uses every session,
and multiple search terms are crawled in parallel.
- Contact scraping follows the site's own contact/about links (loaded in
parallel, first hit wins) instead of walking nine guessed URLs one by one.
Page bodies are capped, dead-end social/aggregator "websites" are skipped, and
each domain is scraped once per run.
- Adaptive pacing: fixed 1.5-4s waits replaced by a per-session delay that speeds
up while Google is happy and backs off on throttling; a 429/403 now rotates the
session onto a fresh IP instead of sleeping 30s.
- Reviews and contacts for a place are fetched concurrently; places with no
reviews no longer trigger a review request.
- Dataset rows are pushed in batches and billing events are charged with counts
(a place with 20 reviews cost 20 separate API calls before).
- Fixed emails picking up trailing escape characters from JSON embedded in page
HTML (
mailto:x@y.com\"), which produced a bogus duplicate entry next to the
real address. Normalized addresses are now validated before being published.
- New
skipDuplicateEmails (on by default): keeps only the first place per
email address. Booking platforms and website templates hang one shared mailbox
(e.g. pomoc.pl@booksy.com) on dozens of listings, which filled result sets
with the same lead. Places with no email are unaffected, and a dropped repeat
frees its slot so maxItems is still filled — with distinct leads.
maxItems — global cap on the total number of places pushed across all search terms and locations. Once reached, the actor stops searching and pushing immediately. maxCrawledPlacesPerSearch still applies per (search term, location) pair.
- Scrape Google Maps places: name, address, phone, website, opening hours, rating, reviews count, coordinates, category, price level, plus/area codes, and open/closed status.
- Contact enrichment: pull emails and social profiles (Facebook, Instagram, LinkedIn, YouTube, TikTok, Telegram, X/Twitter) from each place's website.
- Reviews: attach up to
maxReviewsPerPlace reviews per place (text, rating, author, date, photos) when includeReviews is enabled.
- Images: attach up to
maxImagesPerPlace image URLs per place when includeImages is enabled.
deepSearch to get past Google's ~120-results-per-area limit.skipPlacesWithoutEmail lead-list mode — keep only places that have an email.- Dataset views: Overview and Contact columns.
- Pay-per-event pricing: per place, per contact enrichment, per review, per image.
- Multi-language search and address parsing across 60+ countries with auto-detected ISO-2 country codes.
