Google Maps Reviews Scraper

A production-ready, high-volume Google Maps review scraper built in TypeScript (Node.js 20+) using Crawlee and Playwright (headless Chromium). Given Google Maps place URLs or Place IDs, it extracts all reviews for each place with complete review text, ratings, reviewer profiles, owner responses, photos, translation variants, and metadata, exportable to JSON, NDJSON, CSV, and Excel (.xlsx).

---

Important Compliance Notice (Personal Data & TOS)

[!CAUTION] Reviewer Personal Data & GDPR Compliance: Reviewer data (display name, profile URL, contributor ID, and profile photo) constitutes personal data under privacy regulations such as the General Data Protection Regulation (GDPR), CCPA, and equivalent frameworks worldwide.

• Opt-In Privacy Gating (includeReviewerDetails): By default, reviewer metadata is captured. When includeReviewerDetails is set to false, the scraper sanitizes reviewer-identifiable fields (name: "Anonymous", reviewerId: "", reviewerUrl: "", reviewerPhotoUrl: null), while preserving review metrics (stars, detailed ratings, review text, timestamps, owner responses).
• Legal Responsibility: Users of this software are solely responsible for ensuring that their data collection, storage, and processing practices comply with applicable data protection laws and Google Maps' Terms of Service. This is especially critical before retaining or analyzing reviewer-identifiable records beyond aggregate sentiment or analytical research.

---

Key Features

• Single-Place-In, All-Reviews-Out: Narrower, deeper extraction designed to retrieve complete review histories for specified venues.
• Crawlee + Playwright Engine: Navigates dynamic JavaScript-rendered infinite scroll review panels with humanized scroll steps and random jitter.
• Residential Proxy Rotation & Resilience: Pluggable proxy configuration with session management, bot/CAPTCHA detection, and backoff recovery.
• Fail-Fast Zod Schema Validation: Strict input validation preventing wasted runs; full Zod schema validation for output records.
• Sort & Date Filtering: Supports sorting by newest, mostRelevant, highestRating, and lowestRating. Combines date-range bounding (dateFrom, dateTo) with newest sort.
• Reviews Origin Handling (all vs google): Handles Google's blended view vs. native Google reviews, critical for hotel and travel venues.
• Full Text & Translation Extraction: Expands truncated "Read more" links; captures both original and translated text variants along with language tags.
• Owner Response Capture: Detects and extracts owner responses and response timestamps.
• Isolated Per-Field Extractors: Every DOM extractor is isolated with individual try/catch boundaries—a missing or non-standard field never drops a review.
• Multi-Format Streaming Storage: Exports incrementally to JSON, NDJSON, CSV, and Excel (.xlsx) simultaneously.
• Production Observability: Structured JSON/console logging, progress metrics, and end-of-run yield shortfall summaries.

---

System Architecture

/src
  /config          # Zod input/output schemas and runtime defaults
  /resolve         # URL & Place ID canonicalization (shortlink follow, ChIJ converter)
  /crawlers
    placeCrawler.ts    # Place-level metadata & cookie interstitial handling
    reviewsCrawler.ts  # Infinite scroll loop, sort, origin, and card expansion
  /extractors      # Pure functions: DOM -> typed fields with per-field error isolation
    placeExtractor.ts
    reviewTextExtractor.ts
    reviewerExtractor.ts
    reviewRatingExtractor.ts
    reviewDateExtractor.ts
    reviewResponseExtractor.ts
    reviewMediaExtractor.ts
    reviewContextExtractor.ts
  /storage         # Export adapters (JSON, NDJSON, CSV, Excel XLSX)
  /proxy           # Residential proxy configuration & session manager
  /utils           # Logger, dateParser, humanize jitter delays
  orchestrator.ts  # Batch orchestrator managing Crawlee PlaywrightCrawler
  index.ts         # CLI entrypoint (reviews-scraper)
/tests
  fixtures/        # DOM snapshots (standard, owner response, image-rich, translated, ad)
  schema.test.ts
  extractors.test.ts
  dateParser.test.ts
  urlResolver.test.ts
  storage.test.ts
  scripts/         # Manual integration sanity script

---

Input Configuration Schema

Field	Type	Default	Description
startUrls	string[]	undefined	Array of Google Maps URLs (direct, search, or maps.app.goo.gl shortlinks).
placeIds	string[]	undefined	Array of raw Google Place IDs (ChIJ...). Canonical URLs are constructed automatically.
maxReviews	number	1000	Maximum reviews to retrieve per place. Helps bound runtime and proxy bandwidth.
reviewsSort	enum	'newest'	Sort order: 'newest', 'mostRelevant', 'highestRating', 'lowestRating'.
reviewsOrigin	enum	'all'	'all' (blended view) or 'google' (Google-native only).
dateFrom	string (ISO)	undefined	ISO 8601 start date. Only valid when reviewsSort === 'newest'.
dateTo	string (ISO)	undefined	ISO 8601 end date. Only valid when reviewsSort === 'newest'.
language	string	'en'	Host language parameter (hl=...) for the Google Maps UI.
includeImages	boolean	true	Extract high-resolution photo URLs attached by reviewers.
includeReviewerDetails	boolean	true	When false, anonymizes personal reviewer identifiers.
concurrency	number	2	Number of places crawled concurrently. Low per-place, parallel across places.
proxyConfig.urls	string[]	undefined	Array of residential proxy URLs (http://user:pass@host:port).

Validation Rules Enforced at Startup

1. Target Requirement: At least one startUrls or placeIds must be provided.
2. Date Range Constraint: dateFrom and dateTo are strictly rejected unless reviewsSort is 'newest'. Sorting by rating or relevance renders reviews non-chronologically; applying a date boundary on those sorts would result in silent under-scraping.
3. Date Logic: dateFrom cannot be later than dateTo.
4. Volume Warnings: If maxReviews is unset or exceeds 5000, a warning is logged detailing the memory and proxy consumption tradeoffs.

---

Output Row Schema (ReviewRow)

Each row represents one review, denormalized with place metadata to enable direct CSV/Excel export without nested flattening:

interface ReviewRow {
  // Review Identification & Content
  reviewId: string;
  reviewUrl: string;
  reviewerId: string;
  reviewerUrl: string;
  name: string;
  reviewerNumberOfReviews: number;
  isLocalGuide: boolean;
  reviewerPhotoUrl: string | null;
  text: string | null;
  textTranslated: string | null;
  originalLanguage: string | null;
  translatedLanguage: string | null;
  publishedAtDate: string; // ISO 8601
  publishAt: string;       // Relative string verbatim (e.g. "2 weeks ago")
  likesCount: number;
  stars: number | null;
  reviewOrigin: string;    // "Google" or third-party name

  // Responses, Media & Metadata
  responseFromOwnerDate: string | null;
  responseFromOwnerText: string | null;
  reviewImageUrls: string[];
  reviewDetailedRating: Record<string, number>; // e.g. { Food: 5, Service: 5 }
  reviewContext: Record<string, string>;        // e.g. { "Visited in": "October 2024" }
  visitedIn: string | null;
  isAdvertisement: boolean;

  // Denormalized Place Information
  placeId: string;
  cid: string;
  fid: string;
  title: string;
  categoryName: string;
  categories: string[];
  totalScore: number | null;
  reviewsCount: number;
  location: { lat: number; lng: number };
  address: string | null;
  url: string;

  scrapedAt: string; // ISO 8601
}

---

Core Scraping Strategies & Edge Cases

1. Handling Hotel & Travel Categories (reviewsOrigin)

Google Maps blends third-party partner reviews (e.g., TripAdvisor, Agoda, Booking.com) into hotel reviews under the default "All" view. This causes two issues:

• Some partner reviews cannot be sorted by "Newest".
• Native Google review yields are reduced. Solution: Set reviewsOrigin: "google" to switch the reviews panel to Google-native reviews only.

2. Google UI Review Count vs. Retrievable Yield Shortfall

A venue's displayed review count (e.g., (1,482)) often exceeds the number of reviews Google's infinite scroll panel actually returns (due to filtered spam, shadow-banned reviews, or API-level deduplication). Solution: The scraper tracks this discrepancy and logs an explicit Yield Shortfall summary at the end of every place run without failing the job:

======================================================================
PLACE SCRAPING SUMMARY: The Roasted Bean (ChIJN1t_tDeuEmsRUsoyG83frY4)
- Reviews Requested: 1000
- Reviews Delivered: 890
- Google Total Count: 950
- Yield Shortfall:    60 (Google UI count vs retrievable reviews gap)
- Stop Reason:        end_of_list
- Duration:           42.4s
======================================================================

3. Infinite Scroll Deduplication

Infinite scroll events or retry attempts can cause duplicate cards to re-render in the DOM. Reviews are deduplicated in real-time by their unique reviewId before being written to disk.

4. Shortened URL Resolution (maps.app.goo.gl)

Shortened URLs do not contain coordinate or Place ID information. The scraper unwinds redirects via fast HTTP headers prior to launching the browser context.

---

Installation & Setup

Prerequisites

• Node.js 20+
• npm 10+

# Clone the repository and enter the directory
git clone <repository-url>
cd "Google maps review scraper"

# Install dependencies
npm install

# Install Playwright Chromium browser binary
npx playwright install chromium

# Build TypeScript
npm run build

---

CLI Usage

1. Run with a JSON configuration file

$node dist/index.js run --config input.example.json --out ./results

2. Run with command line arguments

node dist/index.js run \
  --place-ids ChIJN1t_tDeuEmsRUsoyG83frY4 \
  --max-reviews 100 \
  --sort newest \
  --origin google \
  --date-from 2024-01-01T00:00:00.000Z \
  --format json csv xlsx \
  --out ./results

3. Anonymize reviewer details for GDPR compliance

node dist/index.js run \
  --config input.example.json \
  --no-reviewer-details \
  --out ./results

4. CLI Options Reference

Usage: reviews-scraper run [options]

Options:
  -c, --config <path>            Path to JSON configuration file
  -o, --out <path>               Output directory for export files (default: "./results")
  -f, --format <formats...>      Export format(s): json, ndjson, csv, xlsx, all (default: ["json","csv"])
  -u, --urls <urls...>           Google Maps place URLs
  -p, --place-ids <placeIds...>  Google Place IDs (ChIJ...)
  -m, --max-reviews <number>     Maximum reviews to scrape per place
  -s, --sort <sort>              Sort order: newest, mostRelevant, highestRating, lowestRating
  --origin <origin>              Reviews origin: all, google
  --date-from <isoDate>          ISO 8601 start date (only valid with newest sort)
  --date-to <isoDate>            ISO 8601 end date (only valid with newest sort)
  -l, --language <lang>          Host language (e.g. en, fr, de, es)
  --no-images                    Disable review image extraction
  --no-reviewer-details          Gate personal reviewer data (anonymizes name, photo, IDs)
  --concurrency <number>         Number of places to scrape concurrently (default: 2)
  --proxies <proxies...>         List of residential proxy URLs
  -h, --help                     display help for command

---

Proxy Configuration

Residential proxies are recommended for scraping large volumes of reviews. Configure them either via the .env file or in your input JSON:

$cp .env.example .env

Edit .env:

# Comma-separated residential proxy URLs
PROXY_URLS=http://user:pass@geo.iproyal.com:12321,http://user:pass@proxy2.com:12321

# Browser settings
HEADLESS=true
LOG_LEVEL=info

---

Testing

The project includes an extensive test suite that tests schemas, relative date parsing, canonical URL resolution, and DOM extractors against realistic HTML fixtures without hitting live Google Maps servers:

# Run all unit tests
npm test

# Run tests with coverage
npm run test:coverage

Fixtures Tested

• tests/fixtures/review_card_standard.html: Complete standard review with scores and context.
• tests/fixtures/review_card_owner_response.html: Card containing owner reply text and date.
• tests/fixtures/review_card_with_images.html: Card with high-res photo gallery.
• tests/fixtures/review_card_translated.html: Card with Google machine translation and original text.
• tests/fixtures/review_card_advertisement.html: Card flagged with sponsored/ad badges.
• tests/fixtures/place_page_header.html: Place headline with score, count, category, and address.

Manual Integration Test

To run a live sanity test with Chromium against a real place (maxReviews: 20):

$RUN_INTEGRATION_TEST=true npm run test:manual

---

Docker Deployment

Build and run with Docker:

# Build the Docker image
docker build -t google-maps-reviews-scraper .

# Run with local volume mount for results
docker run --rm \
  -v $(pwd)/results:/usr/src/app/results \
  google-maps-reviews-scraper \
  --config input.example.json \
  --format json csv xlsx \
  --out ./results

---

GitHub Actions Scheduled Workflow

An automated workflow is provided at . It can be triggered on a weekly cron schedule (every Monday at 02:00 UTC) or manually via workflow_dispatch with artifact uploads of the generated reviews.