Google Scholar Scraper With Institution & Journal Data

Google Scholar Scraper — Papers, Citations and Institution Data

Google Scholar Scraper With Institution & Journal Data searches academic literature by keyword, Scholar URL, or institution and returns structured paper records — title, authors, venue, year, citations, DOI, open-access status, and abstract snippet — built from OpenAlex (primary) and augmented with Semantic Scholar. Every result also carries journal/venue quality metrics (h-index, impact-equivalent citedness, APC, DOAJ status) by default, and institution-mode results add institution-level stats (h-index, works count, country). Built for researchers, librarians, and research-intelligence teams who need citation data at scale without hand-copying search results.

What is Google Scholar Scraper With Institution & Journal Data?

It's an Apify Actor that turns bulk keyword searches, Scholar URLs, or institution names into structured academic-paper datasets. Rather than scraping scholar.google.com's HTML directly, it queries OpenAlex's Works API as its primary data source and fills remaining slots from the Semantic Scholar Graph API, then normalizes both into one consistent JSON schema — deduplicated by title, sorted, and capped per your input. No Google account, Google Scholar login, or API key of your own is required.

Key features:

• 🔎 Discover papers by free-text topic, a full Google Scholar URL, or an institution (name, ROR ID, or OpenAlex Institution ID)
• 📄 Core identifying fields per paper: title, authors (plus a detailed per-author list with affiliations), venue, year, DOI, citation count
• 🏛️ Institution-level research-output stats (h-index, works count, country, international-collaboration signal) attached when you target an institution
• 📰 Journal/venue quality metrics (h-index, 2-year mean citedness, open-access status, DOAJ listing, article-processing charge) resolved for every result by default
• 🧰 Real filters: open-access / has-PDF / last-5-years, a publication-year range, publication type, and sort by relevance or citation count
• ⬇️ Standard Apify dataset output — export as JSON, CSV, Excel, XML, or RSS directly from the Console

What data can I extract with Google Scholar Scraper With Institution & Journal Data?

Every pushed row carries bibliographic fields, citation-impact fields, and (when applicable) institution and journal enrichment fields — all in one flat JSON object.

Field	Example value	Use case
query	"citizen science biodiversity monitoring"	The research topic or institution name that produced this row
cidCode	"W4401234567"	OpenAlex work ID — this row's canonical ID
didCode	"W4401234567"	Same ID, kept as a document-ID alias
aidCode	"W4401234567"	Same ID, kept as an article-ID alias
resultIndex	0	Row's position within its query's result set (after filtering/sorting)
type	"journal-article"	Normalized publication type (Crossref-derived)
dataSource	"OpenAlex"	Which API produced this row — OpenAlex or Semantic Scholar
title	"Citizen Science Approaches to Biodiversity Monitoring"	Paper title
link	"https://doi.org/10.1234/abcd.5678"	Landing page URL
documentLink	"https://example.edu/pdf/paper.pdf"	Open-access PDF URL, if one exists
fullAttribution	"A. Ng, B. Lee - Ecology Letters, 2023"	Pre-formatted citation string
authors	"A. Ng, B. Lee, C. Osei"	Comma-joined author names (up to 5)
authorsDetailed	[{"name":"A. Ng","authorId":"A5023456789","institutions":["Stanford University"]}]	Per-author name, OpenAlex author ID, and affiliations (up to 5 authors)
venue	"Ecology Letters"	Journal or conference name
year	2023	Publication year
doi	"10.1234/abcd.5678"	Digital Object Identifier
searchMatch	"This study presents a citizen-science framework..."	Reconstructed abstract snippet, truncated to 300 characters
citations	42	Total citation count
citationsLink	"https://api.openalex.org/works?filter=cites:W4401234567"	OpenAlex query URL for papers citing this one
relatedArticlesLink	"https://api.openalex.org/works?filter=ids.openalex:W1|W2"	OpenAlex query URL for related works
openAccessIsOa	true	OpenAlex's real is_oa flag
oaStatus	"gold"	OA status category (gold / green / bronze / closed)
isRetracted	false	Retraction flag
language	"en"	ISO language code
referencedWorksCount	58	Number of works this paper cites
correspondingAuthorIds	["A5023456789"]	OpenAlex IDs of corresponding authors
sustainableDevelopmentGoals	["Life on Land","Climate Action"]	UN SDG tags (top 5)
countsByYear	[{"year":2023,"citations":12},{"year":2022,"citations":20}]	Citation counts by year
citationVelocity	15.33	Average citations/year over the most recent 3 years with data
fwci	1.42	Field-Weighted Citation Impact (OpenAlex)
topics	[{"name":"Biodiversity Monitoring","score":0.87}]	Top 5 OpenAlex topics with scores
concepts	[{"name":"Ecology","score":0.79}]	Top 5 OpenAlex concepts with scores
percentileRank	88.5	This paper's citation-count percentile within its own query's result set
scrapedAt	"2026-08-15T10:04:22.101Z"	UTC timestamp the row was built
institutionName	"Stanford University"	Resolved institution name (institution-mode rows only)
institutionCountry	"US"	Institution's country code
institutionType	"education"	OpenAlex institution type
institutionWorksCount	214830	Institution's total works count
institutionCitedByCount	18452301	Institution's total citation count
institutionHIndex	356	Institution's h-index
institutionI10Index	98234	Institution's i10-index
authorInstitutions	"Stanford University, University of Oxford"	Up to 8 distinct institutions found among this paper's authors
authorInstitutionCountries	"GB, US"	Sorted, unique country codes among authors' institutions
isInternationalCollaboration	true	true when authors' institutions span more than one country
journalHIndex	187	Journal/venue's h-index (OpenAlex Sources)
journal2yrMeanCitedness	6.42	2-year mean citedness — the impact-factor-equivalent metric
journalWorksCount	32450	Total works published in this venue
journalIsOA	false	Whether the venue is fully open access
journalIsInDOAJ	false	Listed in the Directory of Open Access Journals
journalApcUsd	3450	Article-processing charge in USD
journalIssnL	"0000-0000"	Linking ISSN
journalHostOrganization	"Wiley"	Publisher / host organization name

Institution and journal quality signals

When you list an institution under 🏛️ Institution Targets, that target is resolved (by name, ROR ID, or OpenAlex Institution ID) to its full OpenAlex works list, and every paper in it is tagged with institutionName, institutionCountry, institutionHIndex, institutionWorksCount, and isInternationalCollaboration. Independently, with 📰 Attach journal / venue quality metrics on (the default), every result — from a topic search or an institution search — gets its journal resolved once via OpenAlex Sources and cached per run, filling journalHIndex, journal2yrMeanCitedness, journalApcUsd, journalIsOA, and journalIsInDOAJ. A librarian screening submission venues can use journalApcUsd and journalIsInDOAJ together to flag predatory-adjacent open-access charges before a researcher commits to a journal.

Targeting and filtering fields

year, type, openAccessIsOa, and isRetracted are the fields you filter and sort on. Combine publishedFrom/publishedTo for a year window, publicationType to restrict to journal articles, conference papers, books, or preprints, and resultFilter to require a PDF link, a real open-access flag, or the last five years only. rankBy: cited_by_count re-sorts the full fetched-and-filtered set by citations before truncating to your item cap, so "most cited" reflects everything matched, not an arbitrary early page.

Why not build this yourself?

OpenAlex and Semantic Scholar both publish free, public APIs — OpenAlex's Works API and the Semantic Scholar Graph API — so nothing here is behind a paywall you couldn't reach yourself. What this Actor removes is the integration work: cursor-based pagination past OpenAlex's 200-results-per-page ceiling, deduplicating and reconciling two APIs with different schemas into one shape, resolving free-text institution names to canonical OpenAlex/ROR IDs, resolving and caching one journal lookup per unique venue instead of one per paper, and computing derived fields (citation velocity from counts_by_year, an in-set citation percentile) that neither API returns directly. It also auto-escalates from a direct connection through Apify's proxy tiers with retries if either upstream API rate-limits or blocks a request, so a long institution-mode run doesn't just die on one hiccup.

If you only need a handful of lookups against OpenAlex or Semantic Scholar directly, their own APIs are the right tool. If you need bulk, deduplicated, institution- and journal-enriched paper data landed as one dataset — with pagination, retries, and proxy fallback already handled — that's what this Actor is for.

How to use data extracted from Google Scholar?

Researchers and literature-review teams

Set 🔬 Research Topics to your review's search strings, resultFilter to open_access or has_pdf so every row is actually retrievable, and publishedFrom to your review's cutoff year. The dataset lands with title, authors, doi, searchMatch (a real reconstructed abstract, not an inverted-index dump), and citations per paper — ready to import into a reference manager or screening spreadsheet without re-typing citations by hand.

Research-office and library benchmarking

Point 🏛️ Institution Targets at your own or a peer institution (name, ROR ID, or OpenAlex ID) with 📰 journal analytics on, and every paper that institution published comes back tagged with institutionHIndex, institutionWorksCount, and the journal's journalApcUsd and journalIsInDOAJ. Run it against a shortlist of peer institutions on a schedule to track output growth and open-access adoption over time.

Market and research intelligence

Rank by citations (rankBy: cited_by_count) across a set of topics to map which venues and institutions dominate a research area, using venue, journalHIndex, and topics to build a competitive landscape of who publishes where and how often it gets cited.

AI agents and automated pipelines

Feed title, searchMatch, and topics into a RAG index for literature-grounded question answering, or call the Actor from an agent pipeline to enrich a paper list with journalApcUsd and journalIsOA before recommending a submission venue.

⬇️ Input sample

Parameter	Required	Type	Description	Example value
researchTopics	No	Array	Bulk list of search keywords or full Google Scholar URLs, one per line. Provide this OR institutionTargets (or both).	["Tomato Shelf Life Prediction using IoT and Machine Learning"]
searchQueries	No	Array	Legacy alias for researchTopics — kept for backward compatibility.	[]
papersPerTopic	No	Integer	Maximum papers to fetch per topic or institution. Min 1, max 5000.	10
maxItems	No	Integer	Legacy alias for papersPerTopic. Min 1, max 5000, default 10.	10
resultFilter	No	String (enum)	Restrict results by availability/recency: all, has_pdf, open_access (real is_oa flag), recent_5_years.	"open_access"
filter	No	String (enum)	Legacy alias for resultFilter. Default "all".	"all"
publishedFrom	No	Integer	Keep only papers published this year or later. Min 1800, max 2100.	2020
newerThan	No	Integer	Legacy alias for publishedFrom. Min 1800, max 2100.	2020
publishedTo	No	Integer	Keep only papers published this year or earlier. Min 1800, max 2100.	2024
olderThan	No	Integer	Legacy alias for publishedTo. Min 1800, max 2100.	2024
rankBy	No	String (enum)	Sort order: relevance or cited_by_count.	"cited_by_count"
sortBy	No	String (enum)	Legacy alias for rankBy. Default "relevance".	"relevance"
publicationType	No	String (enum)	Restrict to a type: any, journal, conference, book, preprint.	"journal"
articleType	No	String (enum)	Legacy alias for publicationType. Default "any".	"any"
institutionTargets	No	Array	Institution names, ROR IDs, or OpenAlex Institution IDs, one per line. Each is expanded into that institution's full works list with institution stats attached.	["Stanford University"]
enableJournalAnalytics	No	Boolean	Resolve journal/venue quality metrics for every result. Default true.	true
debugMode	No	Boolean	Store a dump of every raw API response in the default key-value store.	false
enableDebugDumps	No	Boolean	Legacy alias for debugMode. Default false.	false
proxyConfiguration	No	Object	Apify proxy config. Default {"useApifyProxy": false} — connects directly and auto-escalates through proxy tiers on a block or rate-limit.	{"useApifyProxy": false}

{
  "researchTopics": [
    "citizen science biodiversity monitoring"
  ],
  "institutionTargets": [],
  "papersPerTopic": 50,
  "resultFilter": "open_access",
  "publishedFrom": 2020,
  "publicationType": "journal",
  "rankBy": "cited_by_count",
  "enableJournalAnalytics": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Common pitfall: the Console form shows 10 as the prefilled value for 📊 Papers per topic, but that is a UI prefill, not a schema default. If you start a run through the API with an input object that omits both papersPerTopic and its legacy alias maxItems entirely, the code falls back to 100 papers per target — not 10. Set papersPerTopic explicitly if you need a small, predictable cap. Also, if you set both a new-style parameter and its legacy alias (e.g. both researchTopics and searchQueries), the new-style key always wins and the legacy value is silently ignored.

⬆️ Output sample

Every result is a flat, typed JSON object — no nesting to unwrap beyond the author/topic/citation-history sub-lists that are genuinely list-shaped. Export it as JSON, CSV, Excel, XML, or RSS from the Apify dataset.

{
  "query": "citizen science biodiversity monitoring",
  "cidCode": "W4401234567",
  "didCode": "W4401234567",
  "aidCode": "W4401234567",
  "resultIndex": 0,
  "type": "journal-article",
  "dataSource": "OpenAlex",
  "title": "Citizen Science Approaches to Biodiversity Monitoring",
  "link": "https://doi.org/10.1234/abcd.5678",
  "documentLink": "https://example.edu/pdf/paper.pdf",
  "fullAttribution": "A. Ng, B. Lee - Ecology Letters, 2023",
  "authors": "A. Ng, B. Lee, C. Osei",
  "authorsDetailed": [
    { "name": "A. Ng", "authorId": "A5023456789", "institutions": ["Stanford University"] }
  ],
  "venue": "Ecology Letters",
  "year": 2023,
  "doi": "10.1234/abcd.5678",
  "searchMatch": "This study presents a citizen-science framework for biodiversity monitoring...",
  "citations": 42,
  "citationsLink": "https://api.openalex.org/works?filter=cites:W4401234567",
  "relatedArticlesLink": "https://api.openalex.org/works?filter=ids.openalex:W111|W222",
  "openAccessIsOa": true,
  "oaStatus": "gold",
  "isRetracted": false,
  "language": "en",
  "referencedWorksCount": 58,
  "correspondingAuthorIds": ["A5023456789"],
  "sustainableDevelopmentGoals": ["Life on Land", "Climate Action"],
  "countsByYear": [
    { "year": 2023, "citations": 12 },
    { "year": 2022, "citations": 20 }
  ],
  "citationVelocity": 15.33,
  "fwci": 1.42,
  "topics": [{ "name": "Biodiversity Monitoring", "score": 0.87 }],
  "concepts": [{ "name": "Ecology", "score": 0.79 }],
  "percentileRank": 88.5,
  "scrapedAt": "2026-08-15T10:04:22.101Z",
  "institutionName": null,
  "institutionCountry": null,
  "institutionType": null,
  "institutionWorksCount": null,
  "institutionCitedByCount": null,
  "institutionHIndex": null,
  "institutionI10Index": null,
  "authorInstitutions": "Stanford University",
  "authorInstitutionCountries": "US",
  "isInternationalCollaboration": false,
  "journalHIndex": 187,
  "journal2yrMeanCitedness": 6.42,
  "journalWorksCount": 32450,
  "journalIsOA": false,
  "journalIsInDOAJ": false,
  "journalApcUsd": 3450,
  "journalIssnL": "0000-0000",
  "journalHostOrganization": "Wiley"
}

The institution* fields stay null on topic-search rows and are only filled when the row came from an 🏛️ Institution Targets entry; the journal* fields are filled whenever 📰 Attach journal / venue quality metrics is on, regardless of discovery mode.

How do you filter and target specific papers?

This is where result precision comes from — combine these four axes rather than relying on one broad keyword.

1. Topic keyword vs. institution target. researchTopics is a free-text or Scholar-URL search — best when you know the subject but not who wrote about it. institutionTargets (name, ROR ID, or OpenAlex Institution ID) instead expands into everything one institution published, with institution stats attached to every row — best when the institution is your unit of analysis, not the topic.
2. Temporal and type scoping. publishedFrom/publishedTo bound the result set to a year range, and publicationType narrows to journal articles, conference papers, books/chapters, or preprints — useful when a review protocol requires excluding preprints or pre-2020 work.
3. Availability and quality signal. resultFilter: open_access keeps only papers where OpenAlex's real is_oa flag is true (not just "has a PDF link", which has_pdf checks separately); combine with enableJournalAnalytics and read journalIsInDOAJ/journalApcUsd downstream to screen out costly or non-indexed venues.
4. Volume control. papersPerTopic (max 5000) caps each topic or institution independently — five topics each get up to their own cap, not a shared pool — and rankBy: cited_by_count sorts the full fetched-and-filtered set before truncation, so a low cap still returns the most-cited matches rather than the first ones fetched.

{ "researchTopics": ["large language model evaluation"], "resultFilter": "recent_5_years", "publicationType": "conference", "papersPerTopic": 200 }

{ "institutionTargets": ["https://ror.org/00f54p054"], "enableJournalAnalytics": true, "papersPerTopic": 500 }

{ "researchTopics": ["antibiotic resistance surveillance"], "publishedFrom": 2021, "publishedTo": 2024, "rankBy": "cited_by_count", "resultFilter": "open_access" }

▶️ Want to try other SimpleAPI scrapers?

Scraper	What it extracts
Goodreads Book Scraper: Review Sentiment Analysis	Book listings with reader reviews and sentiment scoring
Google Play Scraper: Country Pricing & In-App Purchases	App-store listings with per-country pricing and IAP data
Udemy Course Reviews Scraper: Instructor Course Catalog	Course catalog listings with instructor and review data

How to extract Google Scholar data programmatically

This Actor runs like any Apify Actor: start it from the Console, via the apify-client SDK, or on a schedule/webhook, and read the results back from its dataset as JSON.

Python example

from apify_client import ApifyClient

client = ApifyClient("<APIFY_API_TOKEN>")

run = client.actor("<your-username>/google-scholar-scraper-with-institution-and-journal-data").call(
    run_input={
        "researchTopics": ["citizen science biodiversity monitoring"],
        "resultFilter": "open_access",
        "publishedFrom": 2020,
        "papersPerTopic": 50,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["title"], item["citations"], item["journalHIndex"])

Export to spreadsheets or CRM

Export the dataset as CSV or Excel directly from the Apify Console and map title, authors, venue, year, citations, doi, and journalApcUsd straight to spreadsheet columns — no reshaping needed, since the dataset is already flat.

Is it legal to scrape Google Scholar data?

Yes. This Actor's primary data source, OpenAlex, publishes its bibliographic metadata under a CC0 (public-domain) license, and paper metadata — titles, venues, citation counts, publication years — is factual, published bibliographic data, not personal data. The one field that does identify individuals is authors / authorsDetailed (researcher names and, in institution mode, their affiliations) — attribution information the researchers themselves already published on the paper. If you store or reuse author names at bulk scale, GDPR and CCPA still govern that processing; keep a lawful basis for storage and honor deletion/access requests. Consult legal counsel for commercial applications involving bulk storage of personal data.

❓ FAQ

Are retracted papers included or removed?

Included, but flagged. isRetracted is true when OpenAlex records the paper as retracted; the Actor does not filter retracted papers out automatically, so check that field downstream if your use case requires excluding them.

Can I get institution- and journal-level stats along with the papers?

Yes. Set 🏛️ Institution Targets for institution-level stats (institutionHIndex, institutionWorksCount, institutionCountry) on every paper from that institution, and leave 📰 Attach journal / venue quality metrics on (the default) for journal-level stats (journalHIndex, journal2yrMeanCitedness, journalApcUsd) on every result, whichever discovery mode produced it.

How accurate is the citation and impact data?

It reflects OpenAlex (and, for augmented rows, Semantic Scholar) at request time — the same live, third-party citation databases the fields are sourced from. citations, fwci, and journal2yrMeanCitedness update as those upstream databases reindex; for citation counts in a critical publication or grant context, cross-check against the publisher's own metadata.

How many papers can I get per run?

Up to papersPerTopic (or its legacy alias maxItems), capped at 5000, per topic or institution — independently for each one you list. If neither is set in the input at all, the Actor falls back to 100 per target.

How does institution-mode search differ from a keyword search?

An 🏛️ Institution Targets entry is first resolved (by name, ROR ID, or OpenAlex ID) to one canonical institution, then expanded into that institution's entire OpenAlex works list — every result carries that institution's stats. A 🔬 Research Topics entry is a plain relevance search with no institution attached unless the paper's own authors happen to have one.

Does Google Scholar Scraper With Institution & Journal Data work with Claude, ChatGPT, and AI agent frameworks?

Yes, as a standard HTTP endpoint. Any agent framework that can call the Apify API or the apify-client SDK can trigger a run and read the dataset back as JSON — there is no dedicated MCP server for this specific Actor documented at this time.

How does this Actor compare to other Google Scholar scrapers?

No third-party comparison data was available at the time this README was written, so this section describes only what's verifiable from this Actor's own code: it is the only one in this family that resolves both institution-level and journal-level enrichment on every row, with journal lookups cached once per unique venue per run rather than repeated per paper.

Can I use this Actor without a Google Scholar or OpenAlex API key?

Yes. You need only an Apify account. The Actor calls OpenAlex and Semantic Scholar's public endpoints itself — you never need your own API key for either service, and no Google Scholar login is required.

Conclusion

Google Scholar Scraper With Institution & Journal Data turns topic keywords, Scholar URLs, or institution names into structured, enriched paper datasets — citation counts, DOIs, and abstracts alongside institution-level research-output stats and journal-quality metrics, in one consistent JSON schema. It's built for researchers, library and research-office teams, and AI pipelines that need citation data without re-implementing pagination, deduplication, and institution/journal resolution themselves. Start a run from the Apify Console or the apify-client SDK to get your first dataset.