🔍 Google Scholar Scraper

Google Scholar Scraper — Papers, Authors, Citations & DOIs

Google Scholar Scraper turns a keyword or a Google Scholar search URL into structured academic-paper records — title, authors, venue, year, citation count, DOI, open-access status, and PDF link — as clean JSON, no HTML parsing required. It queries the open OpenAlex and Semantic Scholar databases in real time and merges the two into one deduplicated result set per query. Run it from the Apify Console, the API, or the CLI and pull results straight into a spreadsheet, a database, or an LLM pipeline.

What is Google Scholar Scraper?

Google Scholar Scraper is an Apify Actor that returns academic-paper metadata for a list of search queries or Scholar URLs, sourced from OpenAlex (primary) and Semantic Scholar (used to top up a query when OpenAlex alone doesn't reach the requested count). No Google account, Google Scholar login, or API key is required — the Actor authenticates to neither source, since both OpenAlex and Semantic Scholar are open, publicly queryable databases. It's built for researchers doing literature reviews, ML/RAG engineers building citation-aware datasets, and developers who need paper metadata without maintaining their own HTML scraper or upstream API client.

Worth knowing up front: this Actor does not render or parse scholar.google.com search-result pages. It queries OpenAlex's and Semantic Scholar's own APIs and normalizes both into one schema. In practice this means broad, DOI-anchored academic coverage with a stable JSON shape — but it also means results reflect what OpenAlex and Semantic Scholar have indexed, not necessarily an exact mirror of what a live Google Scholar search page shows for the same query.

What Google Scholar paper data is publicly available to scrape?

Everything this Actor returns is already public: OpenAlex and Semantic Scholar publish paper metadata — titles, authors, venues, years, abstracts, citation counts, and open-access flags — through open APIs that need no login. The only genuine gate is on the underlying paper itself, not on the metadata about it.

Data category	Publicly available	Gated behind
Title, authors, venue, year	✅	—
Citation count, citations-by-year, percentile rank	✅	—
DOI and article type	✅	—
Open-access status (oaStatus)	✅	—
Direct PDF link	✅ when the publisher/repository made one open	Publisher paywall — outside this Actor's control
Full abstract text	✅ when OpenAlex holds the abstract index	Some records have no indexed abstract
Topics, concepts, Sustainable Development Goal tags	✅ (OpenAlex only)	Not computed by Semantic Scholar
Author institution affiliations	✅ (OpenAlex only, top 5 authors)	Not returned by Semantic Scholar

Google Scholar Scraper only returns publicly visible bibliographic data — what OpenAlex and Semantic Scholar already expose to anyone. Nothing behind a login wall, and no full-text scraping of paywalled PDFs.

What data can I extract with Google Scholar Scraper?

Each pushed row is one paper, carrying bibliographic identity fields alongside citation and access metrics. Field names below are copied exactly from the Actor's output.

Paper identity & bibliographic fields

Field Name	Description
query	The search query or URL-derived query string this paper was matched against
resultIndex	Position of this paper within its query's final, filtered result set (0-based)
cidCode / didCode / aidCode	The same identifier repeated under three keys — the OpenAlex work ID (or an MD5-derived ID when no source ID exists)
type	Normalized article type (e.g. journal-article, proceedings-article, book-chapter, preprint)
dataSource	Which upstream source produced this row: "OpenAlex" or "Semantic Scholar"
title	Paper title
link	Landing-page URL for the paper
documentLink	Direct PDF/open-access URL, or empty string if none is available
fullAttribution	Pre-built citation string: "Author, Author - Venue, Year"
authors	Comma-separated names of up to the first 5 authors
authorsDetailed	Array of up to 5 author objects (name, authorId, institutions) — OpenAlex rows only; always [] for Semantic Scholar rows
venue	Journal or conference name
year	Publication year
doi	DOI, or null if none is recorded
searchMatch	A text snippet: the reconstructed abstract (OpenAlex) or the title (Semantic Scholar), truncated to 300 characters for OpenAlex rows
citationsLink	A re-fetchable OpenAlex filter URL listing works that cite this paper (OpenAlex rows), or the paper's own URL (Semantic Scholar rows)
relatedArticlesLink	A re-fetchable OpenAlex filter URL for the paper's own related works (up to 25), or empty string if none
language	ISO language code, or null (OpenAlex only)
topics / concepts	Arrays of up to 5 { name, score } objects (OpenAlex only; [] for Semantic Scholar rows)
sustainableDevelopmentGoals	Array of up to 5 UN SDG label strings (OpenAlex only; [] for Semantic Scholar rows)
correspondingAuthorIds	Array of corresponding-author OpenAlex IDs (OpenAlex only; [] for Semantic Scholar rows)
scrapedAt	UTC ISO-8601 timestamp of when the row was produced

Citation & access metrics

Field Name	Description
citations	Total citation count
citationVelocity	Average citations-per-year over the most recent (up to) 3 years with recorded data; null when no yearly breakdown exists (Semantic Scholar rows)
countsByYear	Array of { year, citations } objects, OpenAlex only; null for Semantic Scholar rows
fwci	OpenAlex's Field-Weighted Citation Impact score, or null if OpenAlex hasn't computed one
referencedWorksCount	Number of works this paper cites (OpenAlex only; null for Semantic Scholar rows)
percentileRank	This paper's citation-count percentile (0 = lowest, 100 = highest) relative to the other papers in this same query's final result set — not a global percentile
openAccessIsOa	Boolean — is the paper open access
oaStatus	OpenAlex's open-access status label (e.g. gold, green, bronze, closed); always null for Semantic Scholar rows
isRetracted	Boolean, or null if retraction status isn't known (OpenAlex only)

🤖 Add-on: Need additional research or content data?

For book and review metadata alongside citation-style data, pair this with the Good Reads Book Scraper from Scraper Engine. If your literature review also needs video-based research talks or conference recordings transcribed, the Youtube Video Subtitles Scraper and Youtube Video Summarizer cover that side.

Why not build this yourself?

Google Scholar has no public, official API — Google has never published one for programmatic search access, which is exactly why OpenAlex and Semantic Scholar exist as open alternatives. So this section isn't "us vs. Google's API" — there is no Google API to compare against. The real choice is: call OpenAlex and Semantic Scholar yourself, or let this Actor do it.

Calling both APIs directly means you own: cursor-based pagination across two different pagination models, deduplicating overlapping results by title, deciding when to fall back from OpenAlex to Semantic Scholar (and merging their different field sets into one schema), computing derived metrics like citation velocity and percentile rank yourself, and building retry/backoff logic for when either API rate-limits or blocks your IP. Google Scholar Scraper does all of that in one call: bulk queries, automatic connection-tier escalation (direct → Datacenter → Residential) on blocked responses, source merging with title-based dedup, and a single normalized JSON schema across both sources.

Use the APIs directly if you only need one source and want to keep total control over pagination logic. Use Google Scholar Scraper when you want both sources merged, deduplicated, filtered, and delivered as one dataset per run.

How to use Google Scholar Scraper

Get results in a few steps — no code required to start a run.

1. Open Google Scholar Scraper on the Apify Store and click Try for free (or Run, if you already have it saved).
2. Fill in searchQueries — the only required field. Add one keyword phrase or Google Scholar URL per line.
3. Optionally set maxItems, filter, sortBy, articleType, newerThan/olderThan, or a custom proxyConfiguration.
4. Click Start and watch progress in the run log — it logs per-query page counts, dedup counts, and how many rows were pushed.
5. When the run finishes, open the Dataset tab and export results as JSON, CSV, Excel, or XML, or pull them via the API.

How to scale to bulk paper extraction

searchQueries is an array (stringList editor) — put as many keywords or Scholar URLs as you want, one per line, in a single run. The Actor processes them sequentially, applying the same maxItems, filters, and sort order to each query, and pushes every query's results into the same dataset with a query field on every row so you can split results back out by query afterward. There's no separate "bulk mode" input — the array itself is the bulk mechanism.

What can you do with Google Scholar paper data?

• 🔬 Researchers doing a literature review use title, doi, and documentLink to build a reading list, and citationsLink to trace forward citations without leaving their pipeline.
• 📊 Bibliometrics analysts use citationVelocity and percentileRank to spot papers gaining citation momentum faster than their query cohort, not just papers with the highest raw citations count.
• 🏛️ Grant and funding teams use sustainableDevelopmentGoals and topics to map a research portfolio against SDG-aligned funding priorities.
• 🔓 Open-access advocates use oaStatus and openAccessIsOa to audit how much of a field's literature is freely readable versus paywalled.
• 🤖 AI engineers feed title, searchMatch, authors, and doi into a RAG index or agent tool, using the typed JSON directly as retrieval context without writing an HTML parser.

How does Google Scholar Scraper handle rate limits and blocking?

The Actor starts every request on a direct connection. If a response comes back with a blocking or rate-limit status code (403, 429, 503, and similar), it automatically escalates to a Datacenter proxy tier, and if that also gets blocked, to a Residential proxy tier — which then stays sticky for the rest of the run, retrying up to 3 times before giving up on that request. Each retry waits roughly 1–2 seconds (with jitter) before trying again. You can also supply your own proxyConfiguration up front as an additional connection option.

⚠️ If a single query still fails after exhausting the connection chain, that query is logged as failed and the run moves on to the next query rather than aborting the whole run — you'll see which queries returned 0 results in the run log, not a hard crash.

⬇️ Input

Parameter	Required	Type	Description	Example Value
searchQueries	Yes	array (stringList)	Bulk list of search keywords (e.g. "Tomato Shelf Life Prediction") OR full Google Scholar URLs (e.g. https://scholar.google.com/scholar?q=...). One per line.	["Tomato Shelf Life Prediction using IoT and Machine Learning"]
maxItems	No	integer (min 1, max 5000, default 10)	Maximum number of papers to fetch per query.	50
filter	No	string enum (default "all")	Restrict results by availability or recency. 'Open access only' checks the real open-access status of the paper; 'Has PDF link' checks whether a direct PDF/document link is available (the two are not always the same paper set). Values: all (📚 All results), has_pdf (📄 Has PDF link), open_access (🔓 Open access only), recent_5_years (🕐 Last 5 years only)	"open_access"
newerThan	No	integer (min 1800, max 2100, nullable)	Scrape only articles from this year on. Example: 2020 keeps everything published in 2020 or later.	2020
olderThan	No	integer (min 1800, max 2100, nullable)	Scrape articles up to this year. Example: 2024 keeps everything published in 2024 or earlier.	2024
sortBy	No	string enum (default "relevance")	How to order the final results. 'Most cited first' also influences which pages are fetched, so it isn't limited to reordering a truncated window. Values: relevance (⭐ Relevance (default)), cited_by_count (🔥 Most cited first)	"cited_by_count"
articleType	No	string enum (default "any")	Restrict to a specific article type. Values: any (🌐 Any type), journal (📰 Journal article), conference (🏛️ Conference paper), book (📕 Book / Book chapter), preprint (📝 Preprint)	"journal"
enableDebugDumps	No	boolean (default false)	Store a dump of every API response (OpenAlex + Semantic Scholar) into the default key-value store so you can inspect what was fetched. Useful for troubleshooting.	false
proxyConfiguration	No	object (default {"useApifyProxy": false})	By default the actor connects directly. If a request is blocked or rate-limited, it auto-escalates to a Datacenter proxy, then to a Residential proxy (with 3 retries). Once switched to Residential, that stays sticky for the rest of the run.	{"useApifyProxy": true}

Example input

{
  "searchQueries": [
    "Tomato Shelf Life Prediction using IoT and Machine Learning",
    "https://scholar.google.com/scholar?q=transformer+attention+mechanism"
  ],
  "maxItems": 50,
  "filter": "open_access",
  "newerThan": 2020,
  "olderThan": 2026,
  "sortBy": "cited_by_count",
  "articleType": "journal",
  "enableDebugDumps": false,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

⬆️ Output

Every run produces one dataset row per paper — a typed, normalized JSON object with a consistent schema whether the row came from OpenAlex or Semantic Scholar. Export as JSON, CSV, Excel, or XML from the Dataset tab, or fetch it via the API.

Example output

{
  "cidCode": "w2981234567",
  "didCode": "w2981234567",
  "aidCode": "w2981234567",
  "resultIndex": 0,
  "type": "journal-article",
  "dataSource": "OpenAlex",
  "title": "Tomato Shelf Life Prediction Using IoT Sensors and Machine Learning",
  "link": "https://journals.example.org/articles/10.1234/tsl.2022.001",
  "documentLink": "https://journals.example.org/articles/10.1234/tsl.2022.001.pdf",
  "fullAttribution": "A. Kumar, S. Patel - Journal of Food Engineering, 2022",
  "authors": "A. Kumar, S. Patel",
  "authorsDetailed": [
    { "name": "A. Kumar", "authorId": "A5023456789", "institutions": ["IIT Delhi"] },
    { "name": "S. Patel", "authorId": "A5023456790", "institutions": ["Anand Agricultural University"] }
  ],
  "venue": "Journal of Food Engineering",
  "year": 2022,
  "doi": "10.1234/tsl.2022.001",
  "searchMatch": "This study proposes an IoT-based sensor network combined with a machine learning model to predict tomato shelf life under variable storage conditions...",
  "citations": 47,
  "citationsLink": "https://api.openalex.org/works?filter=cites:w2981234567",
  "relatedArticlesLink": "https://api.openalex.org/works?filter=openalex_id:w2981239999|w2981238888",
  "openAccessIsOa": true,
  "oaStatus": "gold",
  "isRetracted": false,
  "language": "en",
  "referencedWorksCount": 38,
  "correspondingAuthorIds": ["A5023456789"],
  "sustainableDevelopmentGoals": ["Zero Hunger", "Responsible Consumption and Production"],
  "countsByYear": [
    { "year": 2025, "citations": 9 },
    { "year": 2024, "citations": 14 },
    { "year": 2023, "citations": 15 }
  ],
  "citationVelocity": 12.67,
  "fwci": 1.42,
  "topics": [
    { "name": "Postharvest Physiology", "score": 0.91 },
    { "name": "IoT in Agriculture", "score": 0.78 }
  ],
  "concepts": [
    { "name": "Machine learning", "score": 0.85 },
    { "name": "Shelf life", "score": 0.73 }
  ],
  "percentileRank": 88.5,
  "query": "Tomato Shelf Life Prediction using IoT and Machine Learning",
  "scrapedAt": "2026-08-15T10:42:07.123456+00:00"
}

How does it work?

Google Scholar Scraper does not render or scrape scholar.google.com pages. For each query, it fetches paper metadata directly from the OpenAlex Works API (paginated via cursor, up to 200 results per page), and — only if OpenAlex didn't reach the requested maxItems for that query — tops up the remainder from the Semantic Scholar Graph API (up to 100 per request). Both sources' records are normalized into one shared field schema, deduplicated by lowercased title, then filtered by article type, availability, and year range, sorted, and capped to maxItems before being pushed. Every request starts on a direct connection and escalates through Datacenter and Residential connection tiers automatically if the upstream API returns a blocking or rate-limit status. Because both OpenAlex and Semantic Scholar are open databases with no login wall, every field returned is already public — the output schema stays the same regardless of either API's own internal changes.

Integrations

Google Scholar Scraper runs on the Apify platform, so it works with anything that can call the Apify API or use the apify-client SDK.

Calling Google Scholar Scraper programmatically

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_API_TOKEN>")

run = client.actor("your-username/google-scholar-scraper").call(run_input={
    "searchQueries": ["Tomato Shelf Life Prediction using IoT and Machine Learning"],
    "maxItems": 50,
    "sortBy": "cited_by_count",
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["title"], item["citations"])

Works in Go, Ruby, Node.js, cURL — any language that can make an HTTP request to the Apify API.

No-code tools (n8n, Make, LangChain)

In n8n, use the HTTP Request node (or the Apify community node) pointed at the Actor's run endpoint with your searchQueries in the request body, then feed the returned dataset items into downstream nodes. In Make, the Apify app module runs the Actor and iterates dataset items the same way. In LangChain or a custom agent framework, call the Apify API as a standard tool and pass the JSON rows straight in as retrieval or context documents — the field names are stable, so no custom parsing step is needed.

Is it legal to scrape Google Scholar paper data?

Scraping publicly available bibliographic data — paper titles, authors, venues, citation counts, DOIs — is generally lawful in most jurisdictions, and it's the standard use case for OpenAlex and Semantic Scholar, both of which publish their data specifically for open, programmatic reuse. Google Scholar Scraper returns only data these two open databases already expose, with no login bypass and no paywalled full text.

This is bibliographic/academic metadata, not a personal-data scrape of individual profiles — author names appear as public academic attribution (who wrote a given paper), not as the subject of the dataset, so GDPR's profile-scraping framing doesn't apply here the way it would to a social-profile scraper. The relevant considerations are each source's own terms of use and standard database-rights principles around bulk reuse of a compiled database. Consult legal counsel if your use case involves bulk storage of personal data, including author names, at scale.

Frequently asked questions

What Google Scholar paper fields does Google Scholar Scraper return?

The core fields are title, authors, venue, year, citations, and doi, plus open-access status (oaStatus, openAccessIsOa) and a direct documentLink when one exists. See the full field tables above for all 34 output keys.

Does Google Scholar Scraper require a Google account or login?

No. The Actor never authenticates to Google Scholar, OpenAlex, or Semantic Scholar — all three sources return this data to anonymous requests, so no login, cookie, or API key is needed to run it.

How many papers can I extract in one run?

Up to maxItems per query (1–5000, default 10), across as many queries as you list in searchQueries — there's no separate cap on the number of queries in one run.

What happens if a query returns zero results?

The Actor logs "No more results" for that query and moves on — it doesn't fail the run. If OpenAlex returns nothing and Semantic Scholar also returns nothing for that query, 0 rows are pushed for it while other queries in the same run continue normally.

Can I scrape multiple Google Scholar queries at once?

Yes — searchQueries accepts a list, one keyword or URL per line, and every query is processed and pushed into the same dataset run, tagged with its own query field.

Does Google Scholar Scraper work with Claude, ChatGPT, and other AI agent tools?

It isn't exposed through a dedicated MCP server. It's callable as a standard HTTP endpoint via the Apify API from any agent framework or custom tool wrapper, including LangChain and similar orchestration libraries.

How does Google Scholar Scraper differ from calling OpenAlex or Semantic Scholar myself?

It merges both sources into one deduplicated, filtered, sorted dataset per query, with derived fields (citationVelocity, percentileRank) neither API returns on its own, plus automatic connection-tier escalation when a request gets blocked or rate-limited — logic you'd otherwise have to build yourself.

Does Google Scholar Scraper return data in a format LLMs can use directly?

Yes. Typed, normalized JSON with consistent field names across every run — no HTML parsing, no selectors. Pass rows directly to an LLM, index them into a vector store, or feed them to an agent tool.

What happens if the upstream APIs change their response format?

The Actor is maintained, and the output schema is designed to stay stable across upstream changes — normalization happens once, in the Actor, so a field rename inside OpenAlex or Semantic Scholar doesn't automatically break your downstream JSON keys.

Can I use Google Scholar Scraper without managing proxies myself?

Yes — the Actor handles connection-tier escalation (direct → Datacenter → Residential) and retries automatically. You can still supply your own proxyConfiguration if you want to start on a specific proxy group instead of direct.

Which fields work best for AI training data and RAG indexing?

For RAG: index title, searchMatch (abstract snippet), and fullAttribution as retrievable text, with doi and link as citation anchors. For structured training data: citations, year, citationVelocity, and percentileRank are the most consistently populated numeric fields across OpenAlex rows; note that several enrichment fields (countsByYear, topics, concepts, fwci) are null or empty on Semantic Scholar–sourced rows.

Related scrapers

Scraper Name	What it extracts
Good Reads Book Scraper	Book metadata and reader reviews — the closest bibliographic-entity scraper in this account for published-works data beyond academic papers

Your feedback

Found a bug or missing a field? Let us know through the Actor's Issues tab on the Apify Console, or via the Apify Store contact link on the listing page. Bug reports and field requests are what keep this schema accurate as OpenAlex and Semantic Scholar evolve their own APIs.