Google Search Results Scraper: Research Papers & Citation Counts

Scrapes Google search result pages — the same ranked organic results, ads, and AI answers as the base Google Search Results Scraper — and additionally splits out the research-paper metadata Google prints on academic and technical searches: the by <author> · <year> · Cited by <N> line in front of a result's snippet, and Google's own "Scholarly articles for <search>" box (title, first author, citation count, and a direct link to the publisher's article or PDF).

Built for researchers, science journalists, and evidence-led content teams who need to know which sources behind a topic are worth reading and citing.

What you get on a bare run

Both new "include" flags default on and both filters default open, so a plain run returns:

• Every base field, unchanged — organicResults, paidResults, paidProducts, peopleAlsoAsk, relatedQueries, aiOverview, and the rest.
• On each organic result: citationLine, paperAuthor, paperYear, citedByCount, publishedDate, sourceLabel — parsed from text Google already sends on the page, zero extra requests.
• Google's "Scholarly articles" box, if the query has one: scholarlyArticles[] with each paper's title, first author, citation count, and its publisher URL — one extra request per query.

Consumer queries (recipes, shopping, "best running shoes") legitimately return empty paper fields and an empty scholarlyArticles array. That is not a bug — Google itself does not print this data on those pages.

New input fields

Appended after the full 30-field base input block, unchanged from the base:

Field	Type	Default	What it does
includePaperDetails	boolean	true	Splits the by <author> · <year> · Cited by <N> line and the publisher/source label Google already prints in front of a result's snippet into their own fields, instead of leaving them mashed into description. Zero extra requests — this text is already on the page the Actor downloads. Turn off to leave all six row-level fields null.
includeScholarlyArticles	boolean	true	Also fetches Google's "Scholarly articles for <query>" box: title, first author, citation count, and a link unwrapped to the publisher's own article or PDF. The base's default request carries udm=14 (Google's Web-only tab), which strips this box to zero, so this costs one extra GET per query on the plain search page. Off = byte-identical base behaviour, zero extra requests.
minCitations	integer	0	Live filter: keep only organic results and scholarly articles whose printed "Cited by N" is at least this. 0 keeps everything. Above 0, anything with no citation count printed is dropped. Applies only to what this run just fetched — no history, no cross-run comparison.
publishedFromYear	integer	0	Live filter: keep only organic results whose printed publication year is at least this. 0 keeps every year. Reads the year Google prints against the result (not a Google search-restriction parameter — that's what beforeDate/afterDate above are for). Does not apply to scholarlyArticles — Google does not print a year on that block.
onlyQueriesWithPapers	boolean	false	When on, a query whose page produced neither a scholarly-articles box nor a single organic result with a citation line is skipped entirely (not pushed as an empty-fields row). Evaluated on this run's own results only.

Output

Every base output field, unchanged, same name, same nesting, same value the base would produce — including the base's own known quirks (peopleAlsoAsk[].answer always null, relatedQueries synthesised from organic titles, aiOverview permanently null without a working SerpApi key, resultsPerPage actually meaning "organic rows collected so far", etc. — see Honest limits below). Nothing about the base is repaired here.

Added top-level, right before organicResults:

"hasScholarlyArticles": true,          // convenience flag — was the block present on the page at all
"scholarlyArticlesCount": 3,           // length of the (possibly filtered) scholarlyArticles array
"scholarlyArticles": [
  {
    "position": 1,
    "title": "Structure of SARS-CoV-2 spike protein",
    "publisherUrl": "https://pmc.ncbi.nlm.nih.gov/articles/PMC8423807/pdf/main.pdf",
    "firstAuthor": "Zhang",
    "citedByText": "Cited by 341",
    "citedByCount": 341,
    "scholarSearchUrl": "https://scholar.google.com/scholar?q=sars-cov-2+spike+protein+structure&hl=en&as_sdt=0&as_vis=1&oi=scholart"
  }
]

Added inside every item of organicResults (after position/icon):

{
  // ...the base's own title/url/displayedUrl/description/emphasizedKeywords/siteLinks/productInfo/type/position...
  "citationLine":  "by J Zhang · 2021 · Cited by 341",  // or null
  "paperAuthor":   "J Zhang",                             // or null
  "paperYear":     2021,                                  // or null
  "citedByCount":  341,                                   // or null
  "publishedDate": null,          // the bare-date shape of the same span, when it printed a date instead
  "sourceLabel":   "National Institutes of Health (NIH) | (.gov)"  // or null
}

Every added key is present on every row for shape stability. hasScholarlyArticles is false and scholarlyArticlesCount is 0 with scholarlyArticles: [] when includeScholarlyArticles is off or the block was absent on that query. citationLine and publishedDate are never both non-null on the same row — exactly one of them fires, or neither.

How it works

Google prints a by <author> · <year> · Cited by <N> prefix in front of the snippet of academic/technical organic results — even on the Actor's own default udm=14 ("Web tab") request. This variant reads that text directly (div.VwiC3b > span.YrbPuc) instead of leaving it flattened into description, plus the publisher/source label next to the link (span.VuuXrf). Both are free: no extra request, no extra parameter — the base already downloaded this page.

The "Scholarly articles" box is a different story: it does not exist at all on the udm=14 surface. When includeScholarlyArticles is on, the Actor fires one additional request per query — the identical q/gl/hl/uule/lr/tbs the base already built, with udm and num removed — parses only that box (h3.ohqNXb → the article table → each div[aria-level="3"]), and discards the rest of that response. It never merges any other part of that page (no extra PAA, no extra related-searches, no extra organic rows) into the dataset row.

Honest limits

• Yield is query-shaped. Citation lines and the scholarly-articles box only appear on academic and technical searches. A query like "best running shoes" or a recipe search will legitimately come back with every new field null/empty/false — that is Google's own behaviour, not a parsing gap.
• The scholarly-articles box does not paginate. It has been observed to return at most 3 articles per query; this is an observation, not a documented Google limit, and the Actor does not loop to force more.
• Behind the Apify GOOGLE_SERP/RESIDENTIAL proxy ladder specifically — behaviour was engineered against the documented selectors/paths for this surface; if Google reshapes this page the usual way (class-name churn), the row-level regex parsing degrades gracefully to null, but the scholarly-articles block parsing (anchored on h3.ohqNXb) would need a patch.
• English-shaped parsing. The citation-line regex expects Google's English phrasing (by … · YYYY · Cited by N). Non-English hl/gl runs may return null paper fields even on genuinely academic pages — this has not been verified in other languages.
• All of the base's own known defects are preserved unchanged (see the base actor's own documentation): peopleAlsoAsk[].answer is always null, relatedQueries/suggestedResults are synthesised from organic titles rather than scraped from Google's own related-searches block, aiOverview is permanently null unless a working SerpApi key is supplied, organicResults[].productInfo is always {}, and resultsPerPage in the output means "organic rows collected on this page", not the input value.

Input — full field list

The 30 base fields (see the base actor's documentation for the complete list — queries, resultsPerPage, maxPagesPerQuery, aiMode, serpApiKey, perplexitySearch, chatGptSearch, maximumLeadsEnrichmentRecords, leadsEnrichmentDepartments, focusOnPaidAds, countryCode, searchLanguage, languageCode, locationUule, forceExactMatch, site, relatedToSite, wordsInTitle, wordsInText, wordsInUrl, quickDateRange, beforeDate, afterDate, fileTypes, mobileResults, includeUnfilteredResults, saveHtml, saveHtmlToKeyValueStore, includeIcons, proxyConfiguration) unchanged, in the same order, with the same defaults — followed by the five new fields listed above.