Production-Ready Google Maps Scraper (Apify Actor Parity)

A high-throughput, production-ready Google Maps business data scraper built in TypeScript with Crawlee and Playwright.

Replicates and enhances the core functionality of Apify's compass/crawler-google-places actor:

• Bypasses the ~120-result cap using automated geospatial grid tiling and adaptive 4-way recursive subdivision.
• Extracts rich structured place data directly from Google Maps (IDs, categories, opening hours, review distribution, amenities, images, reviews, phone number, and official website URL as listed on Maps).
• Pure Google Maps Extraction (No External Crawling): This version extracts data exclusively from Google Maps. It does not visit external business websites to extract email addresses or social media profiles. The website field is included only when provided directly on the Google Maps listing.
• Multiple deployment modes: CLI tool, scheduled cron daemon, containerized worker, or headless Fastify REST API.
• Anti-blocking & resilience: Proxy rotation, consent interstitial bypass, captcha detection (sorry.google.com), and defensive DOM selectors.
• Pluggable storage: Streamed JSON, flattened CSV, NDJSON, and SQLite.

---

Table of Contents

• Core Crawl Strategy
• System Architecture
• Input Configuration Reference
• Output Schema
• Installation & Quick Start
• CLI Commands
• HTTP API Reference
• Storage Adapters
• Anti-Blocking & Proxy Management
• Docker Deployment
• Testing
• Legal & Compliance Notice

---

Core Crawl Strategy

Google Maps caps any single search view at approximately 120 results. To achieve exhaustive regional coverage:

[Location String] ──▶ [OSM Nominatim] ──▶ [Bounding Box]
                                                │
                                                ▼
                                    [Geospatial Grid Tiler]
                                                │
                       ┌────────────────────────┴────────────────────────┐
                       ▼                                                 ▼
               [Tile 1: Zoom 14]                                 [Tile 2: Zoom 14]
                       │                                                 │
                       ▼                                                 ▼
             [Scroll Results Feed]                             [Scroll Results Feed]
                       │                                                 │
          ┌────────────┴────────────┐                                    │
    Hit ~120 Cap?             Under Cap?                                 │
          │                         │                                    │
          ▼                         ▼                                    ▼
 [Subdivide into 4x]        [Enqueue Places]                   [Enqueue Places]
  (Zoom 15 Quadrants)       (Global Dedup ID)                 (Global Dedup ID)

1. Geocoding: Resolves place names (e.g. "Austin, TX") to geographical bounding boxes via OpenStreetMap's Nominatim API.
2. Spatial Tiling: Subdivides the bounding box into a grid of overlapping coordinate tiles based on the target zoom level.
3. Scroll-to-load: Programmatically scrolls each search feed (div[role="feed"]) until reaching the "You've reached the end of the list" marker or hitting maxCrawledPlacesPerSearch.
4. Global ID Deduplication: Extracts place stubs and enqueues detail targets into Crawlee's RequestQueue with unique keys based on Google internal IDs (!1s0x..., CID, or Place ID).
5. Adaptive Recursive Subdivision: When a tile yields >= 115 results, it automatically branches into 4 smaller sub-quadrants with increased zoom level (up to maxTileDepth), ensuring high-density areas are fully extracted.

---

Input Configuration Reference

Configurable via a JSON file, environment variables, or CLI arguments:

Field	Type	Default	Description
category	string	undefined	Primary business category or keyword (e.g. "Roofing contractor", "Dentist")
location	string	undefined	City, metro, or address (e.g. "Austin, TX", "London, UK")
searchTerms	string[]	[]	Search queries or multiple keywords (e.g. ["roofing contractors in Austin"])
locationQuery	string	undefined	Free-text location alias
startUrls	string[]	[]	Direct search or place detail public Google Maps URLs
placeIds	string[]	[]	Direct Google Place IDs to scrape directly
categories	string[]	[]	Filter places post-scrape to keep only matching categories
maxCrawledPlacesPerSearch	number	100	Place limit per search tile
maxTotalPlaces	number	undefined	Global cap across the entire crawl run
maxReviews	number	5	Number of recent reviews to scrape per place (0 = skip)
reviewsSort	string	"mostRelevant"	Sort order: newest, mostRelevant, highestRating, lowestRating
zoom	number	14	Map zoom level for grid generation (1-21)
language	string	"en"	Language code for Google Maps (hl param)
scrapePlaceDetailPage	boolean	true	When false, exports stubs from feed without visiting detail page
maxImages	number	10	Max high-resolution photo URLs to extract
includeReviewerInfo	boolean	false	Opt-in for reviewer names/profiles (Privacy compliance)
proxyConfig	object	{ urls: [] }	List of proxy URLs (http://user:pass@host:port)
concurrency	number	3	Parallel Playwright browser workers
outputFormat	string	"json"	Export format: json, csv, ndjson, sqlite
exportDir	string	"./results"	Target directory for generated output files
maxTileDepth	number	2	Max recursion depth for adaptive tile subdivision

[!IMPORTANT] Apify Platform Run Options (Timeout: 0s, Memory: 8192 MB, Max Cost: Unlimited):

• Default Run Options: The Actor specification (.actor/actor.json) is configured with timeoutSecs: 0 (unlimited), memoryMbytes: 8192 (8 GB RAM), and memory limits up to 16384 MB.
• Platform Timeout in Apify Console: When running or saving tasks in the Apify Web Console, make sure the Run options match:
  • Timeout: Set to 0 (unlimited execution time).
  • Memory: Set to 8192 MB (unlocks full multi-core CPU allocation and high-throughput browser concurrency).
  • Maximum cost per run: Leave blank / empty for unlimited.
• Blazing-Fast Extraction: Detail pages extract in 2-3 seconds with instant opening hours detection and optimized review collection. Setting maxReviews: 0 or scrapePlaceDetailPage: false can further accelerate discovery when only business cards/leads are needed.

---

Output Schema (CRM-Ready Business Lead)

Every scraped business is returned as a structured, CRM-ready lead record directly from Google Maps:

{
  "business_name": "Example Roofing Co.",
  "category": "Roofing contractor",
  "categories": [
    "Roofing contractor",
    "General contractor",
    "Siding contractor"
  ],
  "full_address": "123 Main Street, Austin, TX 78701, USA",
  "city": "Austin",
  "state": "TX",
  "postal_code": "78701",
  "phone": "+1 512 555 0100",
  "website": "https://example-roofing.com",
  "rating": 4.6,
  "review_count": 84,
  "reviews": [
    {
      "author": "A. Smith",
      "rating": 5,
      "text": "Quick estimate and clear communication.",
      "date": "2 weeks ago",
      "response_from_owner": null
    }
  ],
  "price": "$$",
  "description": "Family-owned roofing and exterior restoration contractor serving the Austin metro area.",
  "images": [
    "https://lh5.googleusercontent.com/p/AF1Qip...=s1600"
  ],
  "images_count": 1,
  "opening_hours": {
    "Monday": "9:00 AM - 5:00 PM",
    "Tuesday": "9:00 AM - 5:00 PM",
    "Saturday": "Closed"
  },
  "additional_info": {
    "Service options": [{ "On-site services": true }]
  },
  "lead_score": 100,
  "lead_priority": "high",
  "contactability": "phone",
  "opportunity_signals": [],
  "data_source": "google_maps",
  "latitude": 30.2672,
  "longitude": -97.7431,
  "google_maps_url": "https://www.google.com/maps/place/?q=place_id:ChIJ...",
  "place_id": "ChIJN1t_tDeuEmsRUsoyG83frY4",
  "cid": "15074921902713971043",
  "scraped_at": "2026-03-31T10:00:00.000Z",
  "source_url": "https://www.google.com/maps/search/roofing+contractors+in+Austin"
}

Lead Scoring & Prioritization

• lead_score (0-100): Quantifies lead quality and completeness based strictly on Google Maps listing data: direct phone (+30), official website listed on Maps (+25), rating >= 4.0 (+15), reviews count >= 10 (+15), and operating hours (+15).
• lead_priority: "high" (score >= 70), "medium" (40-69), or "low" (< 40).
• contactability: Primary communication channel available directly on Maps: "phone", "website", or "none".
• opportunity_signals: Detects growth and outreach signals for agencies and sales teams ("no_website", "missing_phone", "low_rating", "low_review_count", "missing_hours", "unclaimed_listing").

---

Installation & Quick Start

Prerequisites

• Node.js 20+ (Node 22 or 24 recommended)
• npm or pnpm

1. Clone & Install

git clone <repo-url>
cd "1 Google Maps Scraper"
npm install
npx playwright install chromium
npm run build

2. Configure Environment

Copy .env.example to .env:

$cp .env.example .env

Add residential or datacenter proxy endpoints if crawling at scale:

PROXIES=http://user:pass@pr.oxylabs.io:7777,http://user:pass@zproxy.lum-superproxy.io:22225
LOG_LEVEL=info

3. Run Your First Crawl

# Using CLI options directly
node dist/index.js run --term "artisan bakery" --location "Austin, TX" --max 10 --out ./results

# Or using a configuration file
node dist/index.js run --config ./examples/search-location.json

---

CLI Commands

1. scraper run

Executes a scraping run.

node dist/index.js run [options]

Options:
  -c, --config <path>      Path to JSON configuration file
  -o, --out <dir>          Output directory for exported files
  -f, --format <format>    Export format: json, csv, ndjson, sqlite (default: json)
  -t, --term <term>        Search term (e.g. "coffee")
  -l, --location <loc>     Location string (e.g. "Brooklyn, NY")
  -m, --max <number>       Max places per search
  --concurrency <number>   Crawler worker concurrency (default: 3)
  --headless <boolean>     Run headless browser (default: true)
  --dry-run                Validate extractors against local HTML fixtures without network
  --fixtures <dir>         Path to fixture folder (default: ./tests/fixtures)

2. scraper geocode

Inspects coordinates, bounding box, and grid tile count for any location.

$node dist/index.js geocode "Miami Beach, FL" --zoom 14

3. scraper serve

Launches the Fastify HTTP REST API.

$node dist/index.js serve --port 3000

4. scraper schedule

Runs recurring scheduled jobs via cron.

$node dist/index.js schedule --cron "0 0 * * *" --config ./examples/search-location.json

---

HTTP API Reference

When started with node dist/index.js serve, the scraper exposes full run lifecycle management:

POST /runs

Start a crawl run.

POST /runs
Content-Type: application/json

{
  "searchTerms": ["italian restaurant"],
  "locationQuery": "Chicago, IL",
  "maxCrawledPlacesPerSearch": 20,
  "outputFormat": "json"
}

Response (202 Accepted):

{
  "runId": "run_1726508900",
  "status": "RUNNING",
  "exportDir": "d:/.../results/run_1726508900",
  "message": "Crawl run started successfully"
}

GET /runs/:id

Check real-time progress and telemetry.

{
  "runId": "run_1726508900",
  "status": "RUNNING",
  "metrics": {
    "durationSeconds": 42,
    "tilesProcessed": 3,
    "tilesTotal": 12,
    "placesFound": 36,
    "placesScraped": 34,
    "placesFailed": 0,
    "reviewsScraped": 0,
    "blockedRequests": 0,
    "estimatedProxyCostUsd": 0.051
  }
}

GET /runs/:id/results

Download all scraped place results in JSON or streaming NDJSON.

DELETE /runs/:id

Abort a currently running crawl immediately.

---

Storage Adapters

The scraper writes incrementally ("write-as-you-go") so progress is never lost on crash or manual abort:

• JSON (json): Streams newline-delimited records to places.ndjson during the crawl, and flushes formatted places.json array on completion.
• CSV (csv): Generates flattened tabular records in places.csv. If maxReviews > 0, it additionally writes reviews.csv linked by foreign key placeId.
• NDJSON (ndjson): High-throughput streaming format for ingestion into ELT/data pipelines (Snowflake, BigQuery, ClickHouse).
• SQLite (sqlite): Stores normalized records into SQLite database places.db with relational places and reviews tables and indexed place_id/cid.

---

Anti-Blocking & Proxy Management

• Proxy Rotation: Distributes requests evenly across configured proxy pools. Apify Datacenter proxies are configured by default for maximum speed and cost efficiency, with optional Apify Residential Proxy support for aggressive anti-blocking.
• Proxy Cost Minimization (85–90% Bandwidth Savings): Automatically aborts binary image downloads, map vector tiles, video, fonts, and telemetry beacons across all search and detail pages. Image URLs and place metadata are extracted directly from DOM attributes without downloading binary image data over paid proxy bandwidth.
• Cross-Tile Deduplication: Automatically tracks place IDs across overlapping geographical tiles to prevent redundant place detail page loads.
• Stealth Evasions: Automatically strips navigator.webdriver, overrides plugin and language signatures, mocks the Chrome runtime and permissions API, and randomizes desktop viewports.
• Cookie Consent Bypass: Automatically pre-seeds Google consent cookies (SOCS and CONSENT) and intercepts EU cookie consent redirects (consent.google.com) in multiple languages.
• Captcha / Block Detection & Session Retirement: Monitors for sorry.google.com redirects or "unusual traffic" block banners. On trigger, it records telemetry, permanently retires the proxy session, and immediately retries with a fresh proxy IP.
• CI Dry-Run Mode: Run node dist/index.js run --dry-run to validate DOM selectors against offline HTML fixtures in CI without triggering Google bot detection.

---

Docker Deployment

Build and run in a containerized environment with all Playwright Chromium dependencies pre-installed:

# 1. Build image
docker build -t google-maps-scraper:latest .

# 2. Run CLI scraper
docker run --rm -v $(pwd)/results:/app/results google-maps-scraper:latest \
  run --term "orthodontist" --location "Denver, CO" --max 20

# 3. Or run as HTTP API Server
docker run -d -p 3000:3000 -v $(pwd)/results:/app/results google-maps-scraper:latest \
  serve --port 3000

---

Testing

# Run Vitest test suite (extractors, geo tiling, schema validation)
npm test

# Run TypeScript typecheck
npm run typecheck

# Run offline extractor validation
npm run start -- run --dry-run

---

Legal & Compliance Notice

This software is designed to extract publicly accessible information from Google Maps.

• Terms of Service: Users are solely responsible for ensuring compliance with Google's Terms of Service and applicable web scraping regulations in their jurisdiction.
• Personal Data Protection (GDPR / CCPA): Reviewer personal identifiers (names, profile links) are classified as personal data. Under default settings, this scraper anonymizes reviewer information (includeReviewerInfo: false). Reviewer names should only be scraped after explicit opt-in and in compliance with relevant data privacy laws.