Semantic Scholar Scraper: Papers, Citations & Influence

Semantic Scholar Scraper searches the Semantic Scholar corpus of 200M+ papers and returns one clean, flat record per paper, title, abstract, the AI-written TLDR, citation and influential-citation counts, fields of study, open-access PDF link and every cross-reference ID Semantic Scholar holds.

Semantic Scholar publishes this data for programmatic use, so the Actor needs nothing configured at your end: no browser, no proxy, no login. Paste a free API key into apiKey and a large run finishes sooner and more reliably; see Limits.

Agent skill: SKILL.md

https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/semantic-scholar-scraper.md

What you get

Output field	Meaning
paperId, corpusId, url	Semantic Scholar's own identifiers and the paper's S2 page
doi, arxivId, pubmedId, pubmedCentralId, dblpId	Cross-reference IDs, join straight onto Crossref, arXiv, PubMed or DBLP
title, abstract	Title and full abstract
tldr	One-sentence AI summary generated by Semantic Scholar's TLDR model
year, publicationDate, venue	Publication year, exact date where known, and the venue or conference
journalName, journalVolume, journalPages	Journal placement where the record has it
authors, authorIds	Author names in order, plus their S2 author IDs
citationCount, influentialCitationCount, referenceCount	How often it is cited, how often that citation was load-bearing, and how many works it cites
fieldsOfStudy, publicationTypes	Subject classification and record type (JournalArticle, Conference, Review…)
isOpenAccess, openAccessPdfUrl, openAccessStatus, openAccessLicense	Whether a free PDF exists, where, and under which licence
citations	Papers citing this one, only when includeCitations is on
scrapedAt	Run timestamp

A RUN_SUMMARY record in the key-value store holds per-run counts, the filters used, and any query or paper ID that failed.

Common use cases

• Systematic reviews. Sweep a topic, filter by year and citation floor, and export straight to CSV.
• RAG and knowledge bases. title + abstract + tldr + fieldsOfStudy is a ready-made chunk with metadata attached.
• Bibliometrics. Track how citationCount and influentialCitationCount move for a paper set over months.
• Full-text collection. Turn on openAccessOnly and you get a list of directly downloadable PDFs.
• ID reconciliation. Feed in DOIs from another system and get back arXiv, PubMed and DBLP identifiers for the same works.

Quick start

One topic, the 100 most relevant papers:

{
 "searchQueries": ["retrieval augmented generation"]
}

Recent, well-cited, open-access work across three topics, 200 papers in total:

{
 "searchQueries": [
 "retrieval augmented generation",
 "vector database embedding search",
 "mixture of experts language model"
 ],
 "year": "2022-2026",
 "minCitationCount": 25,
 "openAccessOnly": true,
 "maxResults": 200,
 "apiKey": "YOUR_S2_KEY"
}

Enrich a list of DOIs you already have:

{
 "paperIds": ["10.18653/v1/N19-1423", "arXiv:1706.03762", "PMID:19872477"],
 "fields": ["abstract", "tldr", "authors"]
}

Input

Field	Type	Default	Notes
searchQueries	array	[]	Free-text searches, one per entry. No field syntax, no boolean operators, plain keywords, like the semanticscholar.org search box. Each query paginates independently.
paperIds	array	[]	Specific papers. S2 hash, bare or prefixed DOI, arXiv:1706.03762, PMID:19872477, or a doi.org / arxiv.org / semanticscholar.org URL.
fields	array	all seven	Optional fields per record: abstract, tldr, authors, openAccessPdf, fieldsOfStudy, publicationTypes, journal. Core metadata is always fetched.
includeCitations	boolean	false	Add the list of citing papers to each record. Large payload, off by default.
year	string	-	2021, 2015-2020, 2020- or -2015. Search only.
minCitationCount	integer	0	Drop search hits below this citation count.
openAccessOnly	boolean	false	Keep only papers with a downloadable PDF.
maxResults	integer	100	Papers saved by the whole run, searches and lookups together. Shared out between the requests, not given to each. Ceiling 999.
apiKey	string	-	Free key from semanticscholar.org/product/api. Strongly recommended.

At least one of searchQueries or paperIds is required, there is no meaningful "everything" mode. The two combine freely: papers you name are fetched first, the searches then share whatever is left of maxResults, and results are de-duplicated by paperId, so a paper found twice is saved (and charged) once.

Output example

A real record, fetched by DOI:

{
 "paperId": "df2b0e26d0599ce3e70df8a9da02e51594e0e992",
 "corpusId": 52967399,
 "doi": "10.18653/v1/N19-1423",
 "arxivId": "1810.04805",
 "pubmedId": null,
 "pubmedCentralId": null,
 "dblpId": "journals/corr/abs-1810-04805",
 "title": "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding",
 "abstract": "We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. …",
 "tldr": "A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.",
 "year": 2019,
 "publicationDate": null,
 "venue": "North American Chapter of the Association for Computational Linguistics",
 "journalName": null,
 "journalVolume": null,
 "journalPages": "4171-4186",
 "authors": ["Jacob Devlin", "Ming-Wei Chang", "Kenton Lee", "Kristina Toutanova"],
 "authorIds": ["39172707", "1744179", "2544107", "3259253"],
 "citationCount": 118946,
 "influentialCitationCount": 22739,
 "referenceCount": 63,
 "fieldsOfStudy": ["Computer Science"],
 "publicationTypes": ["JournalArticle", "Conference"],
 "isOpenAccess": true,
 "openAccessPdfUrl": "https://doi.org/10.18653/v1/n19-1423",
 "openAccessStatus": "GOLD",
 "openAccessLicense": "CCBY",
 "citations": null,
 "url": "https://www.semanticscholar.org/paper/df2b0e26d0599ce3e70df8a9da02e51594e0e992",
 "scrapedAt": "2026-08-06T12:41:07.882Z"
}

publicationDate is null for a good share of conference papers, Semantic Scholar only has year for them. tldr is present for most computer-science papers and thinner elsewhere, because the TLDR model was trained on that corpus.

Finding a paper ID

You rarely need one, searching is usually easier, but every form below works in paperIds:

What you have	What to paste
A DOI	10.18653/v1/N19-1423, or the same thing prefixed, DOI:10.18653/v1/N19-1423
An arXiv paper	arXiv:1706.03762 or https://arxiv.org/abs/1706.03762
A PubMed record	PMID:19872477 or the pubmed.ncbi.nlm.nih.gov URL
A Semantic Scholar page	The whole URL, or just the 40-character hash at the end

The Actor strips URLs down to the identifier itself, so pasting links straight out of a browser is fine.

API example

curl -X POST "https://api.apify.com/v2/acts/arman-bd~semantic-scholar-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "searchQueries": ["retrieval augmented generation"],
 "year": "2023-2026",
 "minCitationCount": 20,
 "maxResults": 50
 }'

Limits and behaviour

• maxResults bounds the run, not the query. It is the total number of papers the run may save, so it is also the most you can be charged for. The remaining allowance is divided among the requests still to come, so a query that returns little leaves more for the ones behind it. Requests the cap leaves no room for are never issued, and are listed in RUN_SUMMARY.requestsSkipped rather than dropped quietly. A value outside 1-999 is refused before the first request, never read as "no limit".
• Relevance search reaches 999 papers per query and no further. That is why 999 is also the ceiling for maxResults. To go deeper, split the topic into narrower queries or slice it by year.
• Throttling is the constraint when you run without a key. Semantic Scholar limits how fast it will answer, and at a busy moment that can mean a long wait or no answer at all. The Actor paces itself, retries with a growing gap, and gives up on a bounded budget rather than retrying forever. A request that never gets through is named in RUN_SUMMARY.failures with the reason and with however many papers it had already saved, and requestsThrottled counts them; a thin run therefore always says why it is thin. A free key removes most of this, get one before running anything large.
• Retries distinguish fatal from transient. A malformed request and an unknown paper fail immediately, because retrying cannot help. Throttling and server errors back off, honouring a wait hint when one is given.
• A failing query never aborts the run. It is logged and recorded in RUN_SUMMARY.failures; the remaining queries continue. The run only errors out if every request fails.
• Only requested fields are fetched. The Actor asks for exactly the fields it maps, which keeps responses small; an unrecognised entry in fields is warned about and dropped rather than sent.
• Public data only. No credentials beyond the optional key, no personal data, no access-control bypass.

FAQ

Do I need a proxy? No. Proxy configuration is not required to run this Actor.

Do I need a Semantic Scholar account? Not to run it. The free API key is optional, and worth having for large runs: without one a busy moment can leave a query in RUN_SUMMARY.failures.

What happens if a query fails? It is named in RUN_SUMMARY.failures, with the reason and the number of papers it had already saved, and the run continues with the rest. A search that simply matched nothing is listed separately under queriesWithNoMatches, so an empty result is never mistaken for a lost one. The run only errors out when every request failed.

Can I schedule it? Yes. Citation counts move constantly, so re-running a fixed paperIds list on a schedule and diffing citationCount is a common pattern.

What is an "influential" citation? Semantic Scholar's own classifier flags citations where the cited work shapes the citing paper, rather than appearing once in a related-work list. It is usually 5-20% of the total.

Does it fetch full text? No, metadata plus the abstract. openAccessPdfUrl is the direct link when a free PDF exists.

Why is tldr empty for my papers? The TLDR model covers computer science best and thins out in other fields. Nothing is inferred when the API has no TLDR.

Can I integrate it with something else? Yes, Apify API, client libraries, webhooks, scheduled runs, dataset exports (JSON/CSV/Excel) or MCP. Output is structured JSON.