Google Scholar Scraper

Collect public scholarly search results with clear source provenance. The default googleScholar source reads real result cards from scholar.google.com; the separate openAlex source uses the public OpenAlex API and labels every record accordingly.

The Actor uses validated HTTP extraction rather than a browser, so small runs normally finish in seconds. It validates status and content type, detects challenge pages, omits empty values, deduplicates by stable IDs, and never stores CAPTCHA or error pages as records.

What you can collect

Google Scholar records can include:

• Stable ID, rank, page, query position, and Scholar cluster/result IDs
• Title, canonical work URL, resource type, and source domain
• Authors plus public Scholar author profile IDs and URLs
• Publication/venue, publisher, year, and raw publication line
• Scholar snippet, citation count/link, related-article link, version count/link
• Public PDF URL/source, DOI, and available formats
• Search query, requested filters, source/retrieval labels, and ISO scrape time

OpenAlex mode additionally exposes fields available from that API, such as abstracts, external identifiers, open-access status, topics, concepts, institutions, grants, referenced-work count, and location count. These fields are never attributed to Google Scholar.

Modes

• search: run one Scholar or OpenAlex query.
• batch: run 2–20 queries with bounded concurrency and fair interleaving under one global maxItems limit.
• citations: collect works from a numeric Google Scholar cites cluster. This mode uses Google Scholar only.

Example inputs

Single Google Scholar search:

{
  "mode": "search",
  "source": "googleScholar",
  "query": "transformer neural network",
  "maxItems": 10,
  "maxPages": 2,
  "proxyConfiguration": { "useApifyProxy": false }
}

Fair multiple-query search:

{
  "mode": "batch",
  "queries": ["transformer neural network", "quantum error correction"],
  "maxItems": 20,
  "maxPages": 2,
  "maxConcurrency": 2
}

Citation-cluster mode:

{
  "mode": "citations",
  "clusterId": "15154755818357511167",
  "maxItems": 10
}

Explicit OpenAlex mode:

{
  "source": "openAlex",
  "query": "transformer neural network",
  "yearFrom": 2020,
  "maxItems": 10
}

Search controls

Use yearFrom, yearTo, sortBy, resultLanguage, and includeCitations to represent Scholar search controls. The Actor passes supported filters to Scholar and also verifies year bounds before storing records because target behavior can vary. maxItems, maxPages, maxConcurrency, maxRequestRetries, and requestTimeoutSecs are bounded.

Proxy guidance

Direct access, Apify Residential proxy, and custom proxy URLs are supported. Use Residential only when direct cloud access is blocked.

GOOGLE_SERP is intentionally rejected. That proxy group supports HTTP requests to Google Search result pages; it does not support HTTPS tunneling to scholar.google.com or requests to api.openalex.org. For Actors that scrape ordinary Google Search, GOOGLE_SERP remains the preferred low-cost route.

The Actor uses one internally consistent Chrome/Windows HTTP profile and stable proxy session IDs. Explicit challenge pages are not retried or solved; the run fails closed without storing challenge content.

Output contract

The 77-field typed schema is a union of Google Scholar and OpenAlex fields. Every record contains required identity, URL, source, retrieval, query, and timestamp fields. Optional unavailable fields are omitted—never filled with null, empty strings, empty arrays, placeholders, or fabricated values.

source, sourceProvider, and retrieval make provenance unambiguous:

• Google Scholar: Google Scholar, scholar.google.com, google_scholar_html
• OpenAlex: OpenAlex, openalex.org, openalex_api

Performance and limits

Scholar normally returns about ten cards per page. Small direct runs typically complete in a few seconds, while Residential proxy startup and network routing can add latency. Keep concurrency low and use bounded limits. Google Scholar may rate-limit datacenter or frequently reused IPs; use an authorized Residential or custom proxy when needed.

Google Scholar and OpenAlex markup/data can change. The Actor validates known result and empty-state layouts and reports a clear error instead of silently storing malformed records. Public fields vary by work, so citation, DOI, PDF, author profile, abstract, institution, and grant coverage will not be identical across all records.

Local development

npm ci
npm test
npm run lint
npm run validate:dataset
apify run --purge --input-file qa-inputs/minimal.json
node scripts/audit-dataset.mjs storage/datasets/default

Local storage is not uploaded to Apify Cloud. Use apify push, apify call, and then inspect the resulting cloud dataset for deployment verification.