Semantic Scholar Scraper

Pull academic papers, author profiles, and full citation graphs from Semantic Scholar's 200M+ paper corpus — complete with abstracts, DOIs, arXiv IDs, h-index metrics, citation counts, and direct open-access PDF links. Search by keyword, fetch an exact paper by DOI or arXiv ID, or look up an author profile in one run. Built for researchers, systematic-review teams, and data scientists who need a clean, structured scholarly dataset across every discipline without stitching together the public API one page at a time.

Why This Scraper?

• Three ways in, one dataset — keyword search across titles and abstracts, direct fetch by Semantic Scholar paper ID / DOI / arXiv ID / CorpusId / PMID / URL, and author lookup by ID or profile URL. Mix all three in a single run.
• 200M+ papers across 23 fields of study — filter to any combination of Computer Science, Medicine, Biology, Physics, Economics, Mathematics, Law, Linguistics, and 15 more — exact filters, not fuzzy "suggestions", and every row carries the subject labels the filter matched on so you can re-filter the export and get the same set back.
• 12 publication-type filters — narrow to peer-reviewed JournalArticle, Review, MetaAnalysis, ClinicalTrial, Conference, Dataset, Book, and more for systematic-review-grade precision.
• Citation + reference graph expansion — opt in to pull every paper that cites a work, or every paper it references, as separate rows — capped per paper so even "Attention Is All You Need" stays bounded.
• Author profiles with h-index — name, paper count, total citations, and h-index on every author record, plus affiliation and homepage wherever Semantic Scholar holds them, and an opt-in full publication list per author.
• Identifier-rich rows — every row carries a native 40-character paper ID, canonical URL, per-author IDs, and influential-citation count, with a DOI on roughly 19 of every 20 papers and an arXiv ID on every preprint, so you can join against PubMed, Crossref, or arXiv downstream.
• Direct open-access PDF links — openAccessPdfUrl and an isOpenAccess flag surface free full text on every eligible paper, with an open-access-only filter to keep just the downloadable ones.
• High-impact filtering — minimum-citation-count, year-range, and sort-by-citations-or-date controls let you surface the most-cited or most-recent work in a field instantly.
• No API key, no sign-up — go from a keyword or a DOI to a structured dataset of up to 10,000 papers per query.

Use Cases

Literature Reviews & Systematic Reviews

• Assemble a complete, deduplicated reading list for a new topic in minutes
• Filter to Review and MetaAnalysis types for evidence-synthesis projects
• Restrict to open-access PDFs to build a downloadable full-text corpus

Research Trend Analysis

• Track publication volume in a field across a year range
• Surface the most-cited papers of the last two years with citation sorting
• Detect emerging sub-fields from a burst of recent open-access work

Citation Network Mapping

• Expand a seminal paper's citing-paper graph to find follow-up research
• Pull a paper's reference list to trace its intellectual lineage
• Build directed citation edges between papers for bibliometric graphs

Competitive Research Intelligence

• Monitor what a lab or institution is publishing by author ID
• Benchmark researcher output with h-index, paper count, and total citations
• Quantify a topic's influence with influential-citation counts

Academic Lead Generation

• Find domain experts to quote, interview, or recruit via author profiles
• Pull an expert's full publication list to see exactly what they work on today
• Identify rising authors by citation growth in a specific field

Dataset Building for Machine Learning

• Harvest titles + abstracts at scale for NLP and recommendation models
• Build labeled corpora filtered by field of study and publication type
• Collect open-access PDF links for full-text mining pipelines

Getting Started

Basic Keyword Search

The simplest run — one topic, 100 papers:

{
    "searchQueries": ["large language models"],
    "maxResults": 100
}

Filtered Search (Year + Field + Open Access)

Narrow to recent, high-impact, open-access computer science work and sort by citations:

{
    "searchQueries": ["retrieval augmented generation"],
    "yearFrom": 2023,
    "yearTo": 2025,
    "fieldsOfStudy": ["Computer Science"],
    "publicationTypes": ["JournalArticle", "Conference"],
    "openAccessOnly": true,
    "minCitationCount": 25,
    "sortBy": "citationCount",
    "maxResults": 200
}

Direct Fetch with Citation + Reference Graph

Fetch exact papers by DOI and arXiv ID, then pull who cites them and what they reference:

{
    "paperIds": [
        "10.1109/cvpr.2016.90",
        "arXiv:1706.03762",
        "https://www.semanticscholar.org/paper/df2b0e26d0599ce3e70df8a9da02e51594e0e992"
    ],
    "includeCitations": true,
    "includeReferences": true,
    "maxCitationsPerPaper": 50
}

Author Profile Lookup

Pull author profiles by ID or URL, with their full publication lists:

{
    "authorIds": ["1688882", "https://www.semanticscholar.org/author/1751762"],
    "includeAuthorPapers": true,
    "maxCitationsPerPaper": 100
}

To find an author ID, open any Semantic Scholar author page and copy the number after /author/ in the URL, or just paste the whole URL. Semantic Scholar sometimes merges duplicate profiles, so if a record comes back under a different ID than the one you supplied, the returned ID is the current one for that researcher.

Input Reference

Search & Input

Parameter	Type	Default	Description
searchQueries	string[]	["large language models"]	Keywords searched across paper titles and abstracts. Each query produces its own result set.
paperIds	string[]	[]	Fetch exact papers by Semantic Scholar paper ID, DOI, arXiv ID, CorpusId, PMID, or paper URL. One record per paper.
authorIds	string[]	[]	Author IDs (numeric) or full profile URLs. Returns an author-profile record with name, affiliations, h-index, and citation count.
maxResults	integer	100	Maximum papers per search query — an exact cap on what you are charged. Set to 0 for all available results (capped at 10,000 per query). Applies to search queries only — paperIds and authorIds runs return every record you list, and the per-record expansions are capped by maxCitationsPerPaper.

Filters

Filters apply to search queries only, not to directly-fetched papers or authors.

Parameter	Type	Default	Description
yearFrom	integer	null	Only include papers published in this year or later (1900–2100).
yearTo	integer	null	Only include papers published in this year or earlier (1900–2100).
fieldsOfStudy	string[]	[]	Restrict to one or more of 23 research fields (Computer Science, Medicine, Biology, Physics, Economics, and more). Matches Semantic Scholar's own classification, returned on every row as s2FieldsOfStudy.
publicationTypes	string[]	[]	Restrict to one or more of 12 types: Review, JournalArticle, CaseReport, ClinicalTrial, Conference, Dataset, Editorial, LettersAndComments, MetaAnalysis, News, Study, Book.
openAccessOnly	boolean	false	Only return papers with a free, downloadable open-access PDF.
minCitationCount	integer	null	Only return papers cited at least this many times — ideal for surfacing high-impact work.
sortBy	string	"relevance"	"Unsorted (no ranking)" (matches come back in no particular order), "Most cited first" (by citation count), or "Most recent first" (by publication date). Pick a sorted option whenever you cap results and want the top papers.

Output Options

The citation, reference, and author-paper expansions each add one row per child item, which multiplies your result count and cost — leave them off unless you need the full graph.

Parameter	Type	Default	Description
includeAbstracts	boolean	true	Include the abstract text for each paper. Disable to shrink the dataset.
includeReferences	boolean	false	For each paper, also output the papers it cites (its reference list) as separate records.
includeCitations	boolean	false	For each paper, also output the papers that cite it as separate records.
maxCitationsPerPaper	integer	50	Caps how many related papers are pulled per source record (1–1000): citing papers, referenced papers, and an author's publications.
includeAuthorPapers	boolean	false	When you provide author IDs, also output each author's publications as separate paper records, up to maxCitationsPerPaper per author.

Output

Every row carries a recordType field — paper or author — so you can filter cleanly downstream. The dataset ships with two ready-made views: Papers and Author profiles.

Paper (recordType: "paper")

{
    "recordType": "paper",
    "paperId": "2c03df8b48bf3fa39054345bafabfeff15bfd11d",
    "title": "Deep Residual Learning for Image Recognition",
    "abstract": "Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substant...",
    "authors": [
        {"authorId": "39353098", "name": "Kaiming He"},
        {"authorId": "1771551", "name": "X. Zhang"}
    ],
    "year": 2015,
    "publicationDate": "2015-12-10",
    "venue": "Computer Vision and Pattern Recognition",
    "publicationTypes": ["JournalArticle", "Conference"],
    "fieldsOfStudy": ["Computer Science"],
    "s2FieldsOfStudy": ["Computer Science"],
    "citationCount": 237440,
    "referenceCount": 54,
    "influentialCitationCount": 32769,
    "doi": "10.1109/cvpr.2016.90",
    "arxivId": "1512.03385",
    "isOpenAccess": true,
    "openAccessPdfUrl": "https://repositorio.unal.edu.co/bitstream/unal/81443/1/98670607.2022.pdf",
    "url": "https://www.semanticscholar.org/paper/2c03df8b48bf3fa39054345bafabfeff15bfd11d",
    "sourceQuery": null,
    "parentPaperId": null,
    "parentAuthorId": null,
    "relation": null,
    "scrapedAt": "2026-08-28T12:38:28.219Z"
}

Core Fields

Field	Type	Description
recordType	string	Always "paper"
title	string	Paper title
abstract	string	Abstract text (null when abstracts are off or unavailable)
authors	object[]	Authors, each {authorId, name}
year	number	Publication year
publicationDate	string	ISO publication date when available
venue	string	Journal or conference name
publicationTypes	string[]	E.g. ["JournalArticle"]
fieldsOfStudy	string[]	Subject areas as supplied by the publisher. Often empty, and it can name a different area than your filter.
s2FieldsOfStudy	string[]	Semantic Scholar's own subject classification, e.g. ["Computer Science"]. This is the one the field-of-study filter matches on, so it is populated on virtually every row. Use this column when you re-filter the export.

Identifiers

Field	Type	Description
paperId	string	Native 40-character Semantic Scholar paper ID
doi	string	DOI when the publisher registered one (about 94% of papers; null for arXiv-only preprints)
arxivId	string	arXiv ID when the paper has a preprint (null otherwise)
url	string	Canonical Semantic Scholar paper page

Metrics

Field	Type	Description
citationCount	number	Times this paper has been cited
referenceCount	number	Number of references in this paper
influentialCitationCount	number	Semantic Scholar's "influential" citation count

Open Access & Lineage

Field	Type	Description
isOpenAccess	boolean	Whether a free open-access PDF exists
openAccessPdfUrl	string	Direct PDF link when a free full text exists (null when it does not — use openAccessOnly to keep only the rows that have one)
sourceQuery	string	The search query that produced this row (null for direct fetches)
parentPaperId	string	Source paper ID on citation/reference child rows (null on primary rows)
parentAuthorId	string	Source author ID on author-publication child rows (null on primary rows)
relation	string	"citation", "reference", or "authorPaper" on child rows (null on primary rows)
scrapedAt	string	ISO 8601 timestamp

Author Profile (recordType: "author")

{
    "recordType": "author",
    "authorId": "1688882",
    "name": "Yann LeCun",
    "affiliations": ["Facebook", "NYU"],
    "homepage": null,
    "paperCount": 406,
    "citationCount": 269528,
    "hIndex": 139,
    "url": "https://www.semanticscholar.org/author/1688882",
    "scrapedAt": "2026-08-28T12:38:30.663Z"
}

Field	Type	Description
recordType	string	Always "author"
authorId	string	Numeric Semantic Scholar author ID
name	string	Author display name
affiliations	string[]	Affiliations Semantic Scholar has on file. Only a small share of profiles carry any, so an empty list is common.
homepage	string	Personal homepage when Semantic Scholar has one on file. Uncommon — most profiles return null.
paperCount	number	Number of papers attributed to the author
citationCount	number	Total citations across all papers
hIndex	number	Author h-index
url	string	Canonical Semantic Scholar author page
scrapedAt	string	ISO 8601 timestamp

When includeAuthorPapers is on, each author's publications are also emitted as paper rows alongside the profile, so an author's full body of work lands in the Papers view ready to filter and sort.

Tips for Best Results

• Fetch by DOI or arXiv ID for guaranteed exact matches. Keyword search is fuzzy; an identifier like 10.1109/cvpr.2016.90 or arXiv:1706.03762 resolves to exactly one paper — perfect for verifying a known reference. If an identifier is missing from Semantic Scholar's index you get a clear notice for that one input and the rest of the run continues.
• Narrow broad topics with filters. A bare query like "machine learning" returns a flood. Add a yearFrom, a fieldsOfStudy value, and a minCitationCount to surface a tight, high-signal set.
• Use minCitationCount for impact triage. Set it to 50 or 100 to skip preprints and low-impact work when you only want established, well-cited literature.
• Filter to Review and MetaAnalysis for evidence synthesis. These publication types are the backbone of systematic reviews and save hours of manual screening.
• Turn off abstracts on large harvests. Setting includeAbstracts: false shrinks every row and speeds up runs when you only need metadata and metrics.
• Keep maxResults modest when expanding the citation graph. includeCitations and includeReferences multiply rows per paper — pair them with a small maxResults (5–20) and a sensible maxCitationsPerPaper to keep runs predictable.
• Sort by "Most recent first" for monitoring. Re-run a saved query on a schedule with date sorting to catch new publications in your field as they land.

Pricing

From $6.00 per 1,000 results — the lowest-cost way to pull discipline-spanning academic data with citation graphs and author metrics bundled in. Bronze, Silver, and Gold subscribers pay progressively less; the table below shows total cost at each discount tier.

Results	No discount	Bronze	Silver	Gold
100	$0.72	$0.68	$0.64	$0.60
1,000	$7.20	$6.80	$6.40	$6.00
10,000	$72.00	$68.00	$64.00	$60.00
100,000	$720.00	$680.00	$640.00	$600.00

No compute or time-based charges — you pay per result, plus a small fixed per-run start fee. A "result" is any row in the output dataset: a paper, an author profile, or a citing/referenced/author-paper row from the opt-in graph expansions (so enabling those expansions increases your result count). Platform fees depend on your Apify plan.

Integrations

Export data in JSON, CSV, Excel, XML, or RSS. Connect to 1,500+ apps via:

• Zapier / Make / n8n — Workflow automation
• Google Sheets — Direct spreadsheet export
• Slack / Email — Notifications on new results
• Webhooks — Trigger custom APIs on run completion
• Apify API — Full programmatic access

Legal & Ethical Use

This actor is designed for legitimate academic research, bibliometrics, literature review, and market intelligence. Users are responsible for complying with applicable laws and Semantic Scholar's terms of service, including making reasonable-rate requests and respecting content usage rules for any papers or PDFs linked from the dataset. Do not use extracted data for spam, harassment, or any illegal purpose.