Production-Ready Google Maps Scraper (Apify Actor Parity)

A high-throughput, production-ready Google Maps business data scraper built in TypeScript with Crawlee and Playwright.

Replicates and enhances the core functionality of Apify's compass/crawler-google-places actor:

• Bypasses the ~120-result cap using automated geospatial grid tiling and adaptive 4-way recursive subdivision.
• Extracts rich structured place data (IDs, categories, opening hours, review distribution, amenities, images, reviews, and website contact enrichment).
• Multiple deployment modes: CLI tool, scheduled cron daemon, containerized worker, or headless Fastify REST API.
• Anti-blocking & resilience: Proxy rotation, consent interstitial bypass, captcha detection (sorry.google.com), and defensive DOM selectors.
• Pluggable storage: Streamed JSON, flattened CSV, NDJSON, and SQLite.

---

Table of Contents

• Core Crawl Strategy
• System Architecture
• Input Configuration Reference
• Output Schema
• Installation & Quick Start
• CLI Commands
• HTTP API Reference
• Storage Adapters
• Anti-Blocking & Proxy Management
• Docker Deployment
• Testing
• Legal & Compliance Notice

---

Core Crawl Strategy

Google Maps caps any single search view at approximately 120 results. To achieve exhaustive regional coverage:

[Location String] ──▶ [OSM Nominatim] ──▶ [Bounding Box]
                                                │
                                                ▼
                                    [Geospatial Grid Tiler]
                                                │
                       ┌────────────────────────┴────────────────────────┐
                       ▼                                                 ▼
               [Tile 1: Zoom 14]                                 [Tile 2: Zoom 14]
                       │                                                 │
                       ▼                                                 ▼
             [Scroll Results Feed]                             [Scroll Results Feed]
                       │                                                 │
          ┌────────────┴────────────┐                                    │
    Hit ~120 Cap?             Under Cap?                                 │
          │                         │                                    │
          ▼                         ▼                                    ▼
 [Subdivide into 4x]        [Enqueue Places]                   [Enqueue Places]
  (Zoom 15 Quadrants)       (Global Dedup ID)                 (Global Dedup ID)

1. Geocoding: Resolves place names (e.g. "Austin, TX") to geographical bounding boxes via OpenStreetMap's Nominatim API.
2. Spatial Tiling: Subdivides the bounding box into a grid of overlapping coordinate tiles based on the target zoom level.
3. Scroll-to-load: Programmatically scrolls each search feed (div[role="feed"]) until reaching the "You've reached the end of the list" marker or hitting maxCrawledPlacesPerSearch.
4. Global ID Deduplication: Extracts place stubs and enqueues detail targets into Crawlee's RequestQueue with unique keys based on Google internal IDs (!1s0x..., CID, or Place ID).
5. Adaptive Recursive Subdivision: When a tile yields >= 115 results, it automatically branches into 4 smaller sub-quadrants with increased zoom level (up to maxTileDepth), ensuring high-density areas are fully extracted.

---

Input Configuration Reference

Configurable via a JSON file, environment variables, or CLI arguments:

Field	Type	Default	Description
category	string	undefined	Primary business category or keyword (e.g. "Roofing contractor", "Dentist")
location	string	undefined	City, metro, or address (e.g. "Austin, TX", "London, UK")
searchTerms	string[]	[]	Search queries or multiple keywords (e.g. ["roofing contractors in Austin"])
locationQuery	string	undefined	Free-text location alias
startUrls	string[]	[]	Direct search or place detail public Google Maps URLs
placeIds	string[]	[]	Direct Google Place IDs to scrape directly
categories	string[]	[]	Filter places post-scrape to keep only matching categories
maxCrawledPlacesPerSearch	number	100	Place limit per search tile
maxTotalPlaces	number	undefined	Global cap across the entire crawl run
scrapeContactEnrichment	boolean	true	Visits place website to extract emails, phone, and social links
maxReviews	number	5	Number of recent reviews to scrape per place (0 = skip)
reviewsSort	string	"mostRelevant"	Sort order: newest, mostRelevant, highestRating, lowestRating
zoom	number	14	Map zoom level for grid generation (1-21)
language	string	"en"	Language code for Google Maps (hl param)
scrapePlaceDetailPage	boolean	true	When false, exports stubs from feed without visiting detail page
maxImages	number	10	Max high-resolution photo URLs to extract
includeReviewerInfo	boolean	false	Opt-in for reviewer names/profiles (Privacy compliance)
proxyConfig	object	{ urls: [] }	List of proxy URLs (http://user:pass@host:port)
concurrency	number	3	Parallel Playwright browser workers
outputFormat	string	"json"	Export format: json, csv, ndjson, sqlite
exportDir	string	"./results"	Target directory for generated output files
maxTileDepth	number	2	Max recursion depth for adaptive tile subdivision

---

Output Schema (CRM-Ready Business Lead)

Every scraped business is returned as a structured, CRM-ready lead record:

{
  "business_name": "Example Roofing Co.",
  "category": "Roofing contractor",
  "full_address": "123 Main Street, Austin, TX 78701, USA",
  "phone": "+1 512 555 0100",
  "website": "https://example-roofing.com",
  "email": "info@example-roofing.com",
  "rating": 4.6,
  "review_count": 84,
  "reviews": [
    {
      "author": "A. Smith",
      "rating": 5,
      "text": "Quick estimate and clear communication.",
      "date": "2 weeks ago",
      "response_from_owner": null
    }
  ],
  "lead_score": 90,
  "lead_priority": "high",
  "contactability": "email",
  "opportunity_signals": [
    "no_social_profile_detected"
  ],
  "enrichment_status": "completed",
  "data_source": "google_maps",
  "latitude": 30.2672,
  "longitude": -97.7431,
  "google_maps_url": "https://www.google.com/maps/place/?q=place_id:ChIJ...",
  "place_id": "ChIJN1t_tDeuEmsRUsoyG83frY4",
  "opening_hours": {
    "Monday": "9:00 AM - 5:00 PM",
    "Tuesday": "9:00 AM - 5:00 PM",
    "Saturday": "Closed"
  },
  "facebook": null,
  "instagram": null,
  "linkedin": null,
  "twitter": null,
  "scraped_at": "2026-03-31T10:00:00.000Z",
  "source_url": "https://www.google.com/maps/search/roofing+contractors+in+Austin"
}

Lead Scoring & Prioritization

• lead_score (0-100): Quantifies lead quality and readiness for sales outreach based on verified email (+25), direct phone (+20), official website (+20), social presence (+10), rating >= 4.0 (+10), reviews count >= 10 (+10), and operating hours (+5).
• lead_priority: "high" (score >= 70), "medium" (40-69), or "low" (< 40).
• contactability: Primary communication channel: "email", "phone", "website", or "none".
• opportunity_signals: Detects growth signals for agencies and sales teams ("no_website", "no_email_detected", "no_social_profile_detected", "low_rating", "low_review_count", "missing_phone", etc.).

---

Installation & Quick Start

Prerequisites

• Node.js 20+ (Node 22 or 24 recommended)
• npm or pnpm

1. Clone & Install

git clone <repo-url>
cd "1 Google Maps Scraper"
npm install
npx playwright install chromium
npm run build

2. Configure Environment

Copy .env.example to .env:

$cp .env.example .env

Add residential or datacenter proxy endpoints if crawling at scale:

PROXIES=http://user:pass@pr.oxylabs.io:7777,http://user:pass@zproxy.lum-superproxy.io:22225
LOG_LEVEL=info

3. Run Your First Crawl

# Using CLI options directly
node dist/index.js run --term "artisan bakery" --location "Austin, TX" --max 10 --out ./results

# Or using a configuration file
node dist/index.js run --config ./examples/search-location.json

---

CLI Commands

1. scraper run

Executes a scraping run.

node dist/index.js run [options]

Options:
  -c, --config <path>      Path to JSON configuration file
  -o, --out <dir>          Output directory for exported files
  -f, --format <format>    Export format: json, csv, ndjson, sqlite (default: json)
  -t, --term <term>        Search term (e.g. "coffee")
  -l, --location <loc>     Location string (e.g. "Brooklyn, NY")
  -m, --max <number>       Max places per search
  --concurrency <number>   Crawler worker concurrency (default: 3)
  --headless <boolean>     Run headless browser (default: true)
  --dry-run                Validate extractors against local HTML fixtures without network
  --fixtures <dir>         Path to fixture folder (default: ./tests/fixtures)

2. scraper geocode

Inspects coordinates, bounding box, and grid tile count for any location.

$node dist/index.js geocode "Miami Beach, FL" --zoom 14

3. scraper serve

Launches the Fastify HTTP REST API.

$node dist/index.js serve --port 3000

4. scraper schedule

Runs recurring scheduled jobs via cron.

$node dist/index.js schedule --cron "0 0 * * *" --config ./examples/search-location.json

---

HTTP API Reference

When started with node dist/index.js serve, the scraper exposes full run lifecycle management:

POST /runs

Start a crawl run.

POST /runs
Content-Type: application/json

{
  "searchTerms": ["italian restaurant"],
  "locationQuery": "Chicago, IL",
  "maxCrawledPlacesPerSearch": 20,
  "outputFormat": "json"
}

Response (202 Accepted):

{
  "runId": "run_1726508900",
  "status": "RUNNING",
  "exportDir": "d:/.../results/run_1726508900",
  "message": "Crawl run started successfully"
}

GET /runs/:id

Check real-time progress and telemetry.

{
  "runId": "run_1726508900",
  "status": "RUNNING",
  "metrics": {
    "durationSeconds": 42,
    "tilesProcessed": 3,
    "tilesTotal": 12,
    "placesFound": 36,
    "placesScraped": 34,
    "placesFailed": 0,
    "reviewsScraped": 0,
    "blockedRequests": 0,
    "estimatedProxyCostUsd": 0.051
  }
}

GET /runs/:id/results

Download all scraped place results in JSON or streaming NDJSON.

DELETE /runs/:id

Abort a currently running crawl immediately.

---

Storage Adapters

The scraper writes incrementally ("write-as-you-go") so progress is never lost on crash or manual abort:

• JSON (json): Streams newline-delimited records to places.ndjson during the crawl, and flushes formatted places.json array on completion.
• CSV (csv): Generates flattened tabular records in places.csv. If maxReviews > 0, it additionally writes reviews.csv linked by foreign key placeId.
• NDJSON (ndjson): High-throughput streaming format for ingestion into ELT/data pipelines (Snowflake, BigQuery, ClickHouse).
• SQLite (sqlite): Stores normalized records into SQLite database places.db with relational places and reviews tables and indexed place_id/cid.

---

Anti-Blocking & Proxy Management

• Proxy Rotation: Distributes requests evenly across configured proxy pools. Residential proxies are recommended for Google Maps to avoid datacenter throttling.
• Stealth Evasions: Automatically strips navigator.webdriver, overrides plugin signatures, randomizes desktop viewports, and injects humanized cursor jitter.
• Cookie Consent Bypass: Automatically intercepts and dismisses Google's EU cookie consent redirects (consent.google.com).
• Captcha / Block Detection: Monitors for sorry.google.com redirects or "unusual traffic" block banners. On trigger, it records telemetry, discards the blocked session, and retries with backoff.
• CI Dry-Run Mode: Run node dist/index.js run --dry-run to validate DOM selectors against offline HTML fixtures in CI without triggering Google bot detection.

---

Docker Deployment

Build and run in a containerized environment with all Playwright Chromium dependencies pre-installed:

# 1. Build image
docker build -t google-maps-scraper:latest .

# 2. Run CLI scraper
docker run --rm -v $(pwd)/results:/app/results google-maps-scraper:latest \
  run --term "orthodontist" --location "Denver, CO" --max 20

# 3. Or run as HTTP API Server
docker run -d -p 3000:3000 -v $(pwd)/results:/app/results google-maps-scraper:latest \
  serve --port 3000

---

Testing

# Run Vitest test suite (extractors, geo tiling, schema validation)
npm test

# Run TypeScript typecheck
npm run typecheck

# Run offline extractor validation
npm run start -- run --dry-run

---

Legal & Compliance Notice

This software is designed to extract publicly accessible information from Google Maps.

• Terms of Service: Users are solely responsible for ensuring compliance with Google's Terms of Service and applicable web scraping regulations in their jurisdiction.
• Personal Data Protection (GDPR / CCPA): Reviewer personal identifiers (names, profile links) are classified as personal data. Under default settings, this scraper anonymizes reviewer information (includeReviewerInfo: false). Reviewer names should only be scraped after explicit opt-in and in compliance with relevant data privacy laws.