Google Scholar Scraper: Native Citations & Versions

Google Scholar Scraper — Extract Papers, Citations & Versions as JSON

Google Scholar Scraper: Native Citations & Versions searches academic literature and returns papers, citation metrics, and native Google Scholar identifiers as structured JSON. By default it queries the OpenAlex and Semantic Scholar bibliographic APIs for DOI, open-access status, retraction flags, and citation trend data; an optional Direct Live Scholar Search mode adds a genuine clusterId, real "All N versions" counts, and related-article links pulled straight from Google Scholar's own search results — data no bibliographic API exposes. Every row is typed, normalized JSON — no HTML, no selectors, no parsing. Enter a topic list and you can be indexing citation-ranked papers into a research pipeline within minutes.

---

What is Google Scholar Scraper: Native Citations & Versions?

It is an Apify Actor that turns a list of research topics or Google Scholar URLs into structured paper records. The default engine calls the OpenAlex and Semantic Scholar public APIs directly — no Google Scholar account, cookie, or login of any kind is required, for either engine. It returns:

• 📄 Papers — title, authors, venue, year, DOI, publication type, abstract snippet
• 📈 Citation & impact metrics — citation count, per-year citation trend, citation velocity, percentile rank, field-weighted citation impact
• 🎓 Native Scholar identifiers (opt-in) — real clusterId, "All N versions" counts, related-article links, document-type badges, sourced from a live Google Scholar fetch
• 📤 Export as JSON, CSV, or Excel from the Apify dataset — no proxy management, no HTML parsing required on your side

---

What data does Google Scholar Scraper: Native Citations & Versions collect?

Every result carries four layers of data: the paper's bibliographic record, its citation/impact metrics, its open-access/integrity flags, and — only when the Direct Live Scholar Search is enabled — real Scholar-native identifiers.

Data Type	Key Fields	JSON Field Names
Paper record	title, authors, venue, year, DOI, publication type	title, authors, authorsDetailed, venue, year, doi, type
Citation & impact metrics	citation count, per-year trend, citation velocity, percentile rank, field-weighted impact	citations, countsByYear, citationVelocity, percentileRank, fwci
Open access & integrity	open-access status, PDF link, retraction flag, corresponding authors	openAccessIsOa, oaStatus, documentLink, isRetracted, correspondingAuthorIds
Native Scholar data (live mode)	Scholar cluster ID, version count, related-articles link, document badge	clusterId, versions, versionsLink, relatedArticlesLink, documentType

Need more Google Scholar data?

If you also need citing-network crawls, author-profile pages, or venue-level bibliometrics beyond a single search run, check the Apify Store's academic and research-data scraper category — this Actor is scoped to topic-based paper search plus citation and version tracking, not full author or venue crawling.

---

Why not build this yourself?

Google Scholar has no official public API — Google has never published one, and its search pages actively challenge automated traffic. Building this yourself means maintaining two separate systems: an OpenAlex/Semantic Scholar client that handles cursor pagination, per-source field gaps, and de-duplication across two schemas, and a Google Scholar HTML scraper that has to detect and recover from block pages, rotate connections, and get re-fixed every time Scholar's markup shifts.

This Actor already does both, keeps a single stable output schema across all three sources, and never fabricates a value it can't verify — a blocked Scholar fetch simply adds zero rows instead of guessing. That combination is the ongoing engineering cost you avoid.

---

Why do developers and teams scrape Google Scholar?

For AI engineers and agent builders

Typed paper records with a stable schema (title, authors, searchMatch, topics, doi) drop straight into a RAG index or an agent tool without an HTML-parsing step. Citation-count and open-access fields let an agent filter a retrieved set to peer-reviewed, non-retracted, freely accessible sources before it ever reaches an LLM context window — useful for building a literature-grounded research assistant that cites verifiable, still-valid sources.

For researchers and analysts

Bulk topic search with orderResultsBy: cited_by_count surfaces the most-cited work in a field for a literature review, and isRetracted plus oaStatus let you filter out withdrawn papers before you cite them. countsByYear and citationVelocity support tracking whether a subfield's citation activity is accelerating. All data comes from OpenAlex, Semantic Scholar, and Google Scholar's own public search results — public academic-record scope only.

For developers building research tools

The stable, single-schema output across three data sources (dataSource: "OpenAlex" | "Semantic Scholar" | "Google Scholar (Live)") is a workable base layer for a bibliometrics dashboard, a citation-tracking alert tool, or a scheduled "new papers on this topic" digest — run on an Apify schedule and pipe the dataset into your own store.

---

How to scrape Google Scholar (step by step)

1. Open Google Scholar Scraper: Native Citations & Versions on its Apify Store listing and click Try for free / Start.
2. Provide scholarTopics — one research topic or full Google Scholar URL per line. No other field is required.
3. Set availability, publishedFromYear/publishedToYear, publicationType, and orderResultsBy to shape the OpenAlex/Semantic Scholar results; optionally turn on enableLiveScholarSearch for native Scholar fields.
4. Click Start and watch results stream into the dataset live, per topic.
5. Open the Output tab and export the run as JSON, CSV, or Excel.

What to do when Google Scholar changes its structure

The Actor is maintained, and the output schema stays the same regardless of what changes on the backend — field names and types don't shift on your end. Because the default engine calls OpenAlex's and Semantic Scholar's own stable APIs, most runs are unaffected by anything happening on Google Scholar's search pages; only the opt-in Direct Live Scholar Search touches Scholar's HTML directly, and its failure mode is documented below: it adds zero rows rather than breaking your run.

---

⬇️ Input

None of the 21 input fields are required — leave everything but scholarTopics (or its legacy alias) at its default and the Actor runs with sane defaults.

Parameter	Required	Type	Description	Example Value
scholarTopics	No	array	Bulk list of research topics (e.g. "Tomato Shelf Life Prediction") OR full Google Scholar URLs. One per line.	["Federated learning healthcare"]
searchQueries	No	array	Legacy alias for scholarTopics — kept for backward compatibility. Merged with scholarTopics if both are set.	[]
resultsPerTopic	No	integer (1–5000)	Maximum number of papers to fetch per topic (real enforced cap: 5000). Prefill 25.	50
maxItems	No	integer (1–5000)	Legacy alias for resultsPerTopic. Default 10.	10
availability	No	string enum	Restrict results by availability or recency: all (📚 All results), has_pdf (📄 Has PDF link), open_access (🔓 Genuinely open access, is_oa), recent_5_years (🕐 Last 5 years only). Prefill all.	"open_access"
filter	No	string enum	Legacy alias for availability. Same enum values. Default all.	"all"
publishedFromYear	No	integer (1800–2100)	Keep only results published in this year or later. Applies to OpenAlex/Semantic Scholar results.	2020
newerThan	No	integer (1800–2100)	Legacy alias for publishedFromYear.	null
publishedToYear	No	integer (1800–2100)	Keep only results published in this year or earlier.	2026
olderThan	No	integer (1800–2100)	Legacy alias for publishedToYear.	null
orderResultsBy	No	string enum	How to order results: relevance (⭐ default) or cited_by_count (🔥 Most cited first — fetches a larger pool before ranking). Prefill relevance.	"cited_by_count"
sortBy	No	string enum	Legacy alias for orderResultsBy. Default relevance.	"relevance"
publicationType	No	string enum	Restrict to a type: any, journal, conference, book, preprint. Prefill any.	"journal"
articleType	No	string enum	Legacy alias for publicationType. Default any.	"any"
enableLiveScholarSearch	No	boolean	Adds extra result rows sourced directly from a live Google Scholar search, carrying genuine clusterId/version/related-article data. Best-effort — some topics may return zero extra rows if the fetch is challenged. Default false.	true
liveSearchYearFrom	No	integer (1800–2100)	Native Scholar year-range start applied only to the live search (independent of publishedFromYear).	2021
liveSearchYearTo	No	integer (1800–2100)	Native Scholar year-range end applied only to the live search.	2026
includePatents	No	boolean	Whether the live search includes patent filings (matches Google Scholar's own "include patents" option). Default true.	false
includeCitationEntries	No	boolean	Whether the live search includes citation-only stub entries (papers Scholar knows of only via other work citing them). Default true.	true
enableDebugDumps	No	boolean	Stores a dump of every OpenAlex/Semantic Scholar API response into the default key-value store, for troubleshooting. Default false.	false
proxyConfiguration	No	object	By default the OpenAlex/Semantic Scholar engine connects directly and auto-escalates through proxy tiers only if blocked or rate-limited (sticky once escalated). The Direct Live Scholar Search always uses its own dedicated connection strategy, regardless of this setting. Default {"useApifyProxy": false}.	{"useApifyProxy": false}

Example JSON input

{
  "scholarTopics": [
    "Tomato Shelf Life Prediction using IoT and Machine Learning",
    "Federated learning healthcare"
  ],
  "resultsPerTopic": 50,
  "availability": "open_access",
  "publishedFromYear": 2020,
  "publishedToYear": 2026,
  "orderResultsBy": "cited_by_count",
  "publicationType": "journal",
  "enableLiveScholarSearch": true,
  "liveSearchYearFrom": 2021,
  "liveSearchYearTo": 2026,
  "includePatents": false,
  "includeCitationEntries": true,
  "enableDebugDumps": false,
  "proxyConfiguration": { "useApifyProxy": false }
}

Common pitfall: the "legacy alias" fields (searchQueries, maxItems, filter, sortBy, articleType, newerThan, olderThan) exist only for backward compatibility with older integrations. Fill in the current field name (scholarTopics, resultsPerTopic, availability, orderResultsBy, publicationType, publishedFromYear, publishedToYear) — if both a field and its legacy alias are set, the current field wins.

---

⬆️ Output

Results are pushed to the Apify dataset as typed, normalized JSON — one row per paper, per topic. Export as JSON, CSV, or Excel from the Output tab, or pull them with the API/SDK. Every row, regardless of source, carries the same 42 keys — fields a source doesn't support are left honestly null or empty rather than fabricated.

Scraped paper — OpenAlex / Semantic Scholar engine

{
  "query": "Tomato Shelf Life Prediction using IoT and Machine Learning",
  "cidCode": "W4409060190",
  "didCode": "W4409060190",
  "lidCode": null,
  "aidCode": "W4409060190",
  "resultIndex": 0,
  "type": "journal-article",
  "dataSource": "OpenAlex",
  "title": "Tomato Shelf Life Prediction using IoT and Machine Learning",
  "link": "https://doi.org/10.1109/iciset62123.2024.10939467",
  "documentLink": "",
  "documentType": "",
  "fullAttribution": "Nazmul Arafin Naim, Raisul Islam - IEEE, 2024",
  "authors": "Nazmul Arafin Naim, Raisul Islam",
  "authorsDetailed": [
    { "name": "Nazmul Arafin Naim", "authorId": "A5012345678", "institutions": ["BRAC University"] }
  ],
  "venue": "IEEE",
  "year": 2024,
  "searchMatch": "This paper presents an IoT and machine learning based system for predicting tomato shelf life...",
  "citations": 1,
  "citationsLink": "https://api.openalex.org/works?filter=cites:W4409060190",
  "relatedArticlesLink": "https://api.openalex.org/works?filter=openalex_id:W4412345678",
  "versions": null,
  "versionsLink": null,
  "doi": "10.1109/iciset62123.2024.10939467",
  "openAccessIsOa": false,
  "oaStatus": "closed",
  "isRetracted": false,
  "language": "en",
  "referencedWorksCount": 28,
  "correspondingAuthorIds": ["A5012345678"],
  "sustainableDevelopmentGoals": ["Zero Hunger"],
  "countsByYear": [{ "year": 2025, "citations": 1 }],
  "citationVelocity": 1.0,
  "fwci": 0.42,
  "topics": [{ "name": "Food Science", "score": 0.91 }],
  "concepts": [{ "name": "IoT Systems", "score": 0.77 }],
  "authorAffiliations": "Nazmul Arafin Naim (BRAC University)",
  "percentileRank": 87.5,
  "clusterId": null,
  "sidebarLinks": null,
  "liveSearchFiltersApplied": null,
  "scrapedAt": "2026-08-15T10:00:00Z"
}

Scraped paper — Direct Live Scholar Search

{
  "query": "Tomato Shelf Life Prediction using IoT and Machine Learning",
  "cidCode": "9182736450192837465",
  "didCode": "1029384756102938475",
  "lidCode": "8172635401928374650",
  "aidCode": "9182736450192837465",
  "resultIndex": 0,
  "type": "PDF",
  "dataSource": "Google Scholar (Live)",
  "title": "Tomato Shelf Life Prediction using IoT and Machine Learning",
  "link": "https://ieeexplore.ieee.org/document/10939467",
  "documentLink": "https://www.researchgate.net/publication/tomato-shelf-life.pdf",
  "documentType": "PDF",
  "fullAttribution": "NA Naim, R Islam - 2024 IEEE Conference, 2024 - ieeexplore.ieee.org",
  "authors": "NA Naim, R Islam",
  "authorsDetailed": [],
  "venue": "2024 IEEE Conference",
  "year": 2024,
  "searchMatch": "This paper presents an IoT and machine learning based system for predicting tomato shelf life...",
  "citations": 1,
  "citationsLink": "https://scholar.google.com/scholar?cites=9182736450192837465",
  "relatedArticlesLink": "https://scholar.google.com/scholar?q=related:9182736450192837465",
  "versions": 3,
  "versionsLink": "https://scholar.google.com/scholar?cluster=9182736450192837465",
  "doi": null,
  "openAccessIsOa": true,
  "oaStatus": null,
  "isRetracted": null,
  "language": null,
  "referencedWorksCount": null,
  "correspondingAuthorIds": [],
  "sustainableDevelopmentGoals": [],
  "countsByYear": null,
  "citationVelocity": null,
  "fwci": null,
  "topics": [],
  "concepts": [],
  "authorAffiliations": null,
  "percentileRank": 87.5,
  "clusterId": "9182736450192837465",
  "sidebarLinks": "https://www.researchgate.net/publication/tomato-shelf-life.pdf",
  "liveSearchFiltersApplied": "years:2021-2026; patents:excluded; citation-entries:included",
  "scrapedAt": "2026-08-15T10:02:11Z"
}

clusterId, versions, versionsLink, and sidebarLinks are only ever real, non-null values on rows where dataSource is "Google Scholar (Live)" — OpenAlex and Semantic Scholar have no equivalent concept, so those fields stay honestly null there rather than being faked. Conversely, fwci, isRetracted, topics, concepts, countsByYear, and sustainableDevelopmentGoals are populated only on OpenAlex-sourced rows.

---

How does Google Scholar Scraper: Native Citations & Versions compare to other scrapers?

Feature	This Actor	Generic single-source alternative
Data sources per run	✅ Up to three (OpenAlex, Semantic Scholar, live Google Scholar) merged into one schema	❌ Typically one source
Native Scholar clusterId / version count	✅ Real values when live search succeeds	❌ Not available without a live Scholar fetch
Output format	✅ Typed, normalized JSON, same 42 keys across all sources	Varies — often raw HTML or source-specific shapes
Null-vs-fabricated policy	✅ Fields the source can't support are left null, never guessed	Not documented
Connection handling	✅ Auto-escalating proxy chain (direct → datacenter → residential), separate chain for live search	Varies

If you're building an AI agent or RAG pipeline, the output-format row is the decision-maker — parsing HTML inside an agent loop is a reliability failure mode, not a feature.

---

How many results can you scrape with Google Scholar Scraper: Native Citations & Versions?

resultsPerTopic (and its legacy alias maxItems) accepts 1 to 5000 per topic — the Actor clamps any input to that range. OpenAlex results are fetched via cursor pagination, up to 200 results per page; if OpenAlex doesn't fill the requested count, the remaining gap is filled from Semantic Scholar, up to 100 results per request. When orderResultsBy is cited_by_count, the Actor fetches a larger pool first — up to min(resultsPerTopic × 3, 2000) results — ranks the whole pool by citation count, then truncates to resultsPerTopic; this means a "most cited first" run above roughly 667 requested results won't rank across the full 5000-item ceiling, only across the first 2000 fetched. The Direct Live Scholar Search, when enabled, adds up to resultsPerTopic extra rows per topic on top of the OpenAlex/Semantic Scholar results — fewer if a fetch is challenged, since it never fabricates rows to reach the target.

---

Integrate Google Scholar Scraper: Native Citations & Versions and automate your workflow

Google Scholar Scraper: Native Citations & Versions works with any language or tool that can send an HTTP request.

REST API integration

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

run = client.actor("<YOUR_USERNAME>/google-scholar-scraper-native-citations-and-versions").call(
    run_input={
        "scholarTopics": ["Federated learning healthcare"],
        "resultsPerTopic": 50,
        "orderResultsBy": "cited_by_count",
        "enableLiveScholarSearch": True,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["title"], item["citations"], item["clusterId"])

Works in Python, Node.js, Go, Ruby, cURL.

MCP for AI agents

This Actor is reachable through Apify's official MCP server at https://mcp.apify.com — connect an MCP-compatible client (Claude, VS Code, or any MCP client) via OAuth or an Authorization: Bearer <APIFY_TOKEN> header, then have your agent search for and add google-scholar-scraper-native-citations-and-versions as a tool mid-conversation and call it directly.

Automation platforms (n8n, Make, LangChain)

In n8n, use the official Apify node, point it at this Actor's ID, and pass the same input fields as above. In Make, the Apify app's "Run an Actor" module works the same way. For LangChain or any custom agent framework, call the Actor through apify_client (or a raw HTTP POST to the run-sync endpoint) and feed the returned JSON rows directly into your chain — no parsing step required.

---

Is it legal to scrape Google Scholar?

Yes — scraping publicly available academic metadata is generally lawful, and this Actor only returns publicly accessible data: the OpenAlex and Semantic Scholar engines query open, public bibliographic APIs, and the optional Direct Live Scholar Search reads Google Scholar's own public search results page. This is business/catalogue-style bibliographic data (paper titles, venues, citation counts, DOIs), not personal profile data, so terms-of-service and database-rights considerations apply rather than GDPR/CCPA — author names here are already-published academic attribution, not scraped personal profiles. Consult legal counsel for commercial use cases involving bulk personal data.

---

Frequently asked questions

Does Google Scholar Scraper: Native Citations & Versions work without a Google Scholar account?

Yes. Neither engine requires a login. The default engine calls the OpenAlex and Semantic Scholar public APIs directly; the optional Direct Live Scholar Search also runs without any account or cookie.

How often is the scraped data updated?

Every run fetches live — there is no caching layer. Each call to OpenAlex, Semantic Scholar, and (if enabled) Google Scholar happens fresh for that run.

What happens when a search topic returns no papers?

The Actor logs it and moves to the next topic; no rows (and no charge) are produced for a topic with zero matches. If one topic in a batch throws an error, it's caught and logged per-topic — the run continues processing the remaining topics rather than aborting.

Can I get full-text PDFs for paywalled papers?

No. documentLink is only populated when a genuine open-access PDF link or Scholar sidebar link exists in the source data — there is no paywall bypass.

How is this Actor billed?

Pay-per-event: one row_result charge per paper row actually pushed to the dataset, for rows from any of the three sources. Exact pricing is set in the Actor's Console listing.

Does Google Scholar Scraper: Native Citations & Versions work for AI agent workflows and LLM pipelines?

Yes. It's reachable as an HTTP endpoint by any agent framework, or through Apify's MCP server (https://mcp.apify.com) as a callable tool. Every response is typed JSON — no parsing step before passing it to an LLM.

How does Google Scholar Scraper: Native Citations & Versions handle Google Scholar's anti-bot system?

The OpenAlex/Semantic Scholar engine starts with a direct connection and escalates through a datacenter, then a sticky residential proxy tier, only on a blocked or rate-limited response (retrying up to 3 times on the residential tier). The Direct Live Scholar Search uses its own dedicated chain — a Google-SERP-tuned proxy tier first, escalating to a sticky residential tier on a genuine block, detected by a confirmed block-phrase match paired with a response-size check so a real large results page is never mistaken for a block page.

How does Google Scholar Scraper: Native Citations & Versions compare to other Google Scholar scrapers?

Unlike a scraper built on a single data source, this Actor merges up to three sources (OpenAlex, Semantic Scholar, live Google Scholar) into one consistent 42-field schema, and marks fields honestly null when a source has no equivalent concept rather than fabricating a value.

Does Google Scholar Scraper: Native Citations & Versions return data in a format LLMs can use directly?

Yes. Typed, normalized JSON with stable field names across every row. Pass it directly into an LLM context window, index it into a vector store, or route it through an agent tool call.

Can I use Google Scholar Scraper: Native Citations & Versions without managing proxies?

Yes. Both connection chains — the OpenAlex/Semantic Scholar engine's and the Direct Live Scholar Search's — are handled automatically, including escalation and retries. Supplying your own proxyConfiguration is optional and only affects the OpenAlex/Semantic Scholar engine.

What happens when Google Scholar changes its structure or blocks the scraper?

The Actor is maintained and the output schema stays stable — field names and types don't change on your end. A structural change to Google Scholar's live search page can affect only the opt-in Direct Live Scholar Search, whose documented failure mode is to add zero extra rows for the affected topic rather than break the run or return fabricated data.

---

Your feedback

Found a bug, or missing a field you need from OpenAlex, Semantic Scholar, or a live Scholar page? We want to know. Use the Issues tab on this Actor's Apify Store page, or reach out through Apify Console support — active maintenance is how the schema stays stable for you.