# Semantic Scholar Scraper: Papers, Citations & Influence (`arman-bd/semantic-scholar-scraper`) Actor Search 200M+ papers on Semantic Scholar and get clean metadata: abstracts, AI-written TLDR summaries, citation and influential-citation counts, open-access PDF links and DOI/arXiv/PubMed IDs. One capped, de-duplicated run. - **URL**: https://apify.com/arman-bd/semantic-scholar-scraper.md - **Developed by:** [Arman Hossain](https://apify.com/arman-bd) (community) - **Categories:** Developer tools, AI, MCP servers - **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.37 / 1,000 paper scrapeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Semantic Scholar Scraper: Papers, Citations & Influence ![Semantic Scholar: Titles, abstracts, AI-written TLDRs, citation counts and cross-reference IDs, one flat record per paper](https://api.apify.com/v2/key-value-stores/ZQOcNAOHrIgTacAmy/records/semantic-scholar-scraper.jpg) **Semantic Scholar Scraper** searches the Semantic Scholar corpus of 200M+ papers and returns one clean, flat record per paper, title, abstract, the AI-written TLDR, citation and influential-citation counts, fields of study, open-access PDF link and every cross-reference ID Semantic Scholar holds. Semantic Scholar publishes this data for programmatic use, so the Actor needs nothing configured at your end: no browser, no proxy, no login. Paste a free API key into `apiKey` and a large run finishes sooner and more reliably; see Limits. **Agent skill: [SKILL.md](https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/semantic-scholar-scraper.md)** ``` https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/semantic-scholar-scraper.md ``` ### What you get | Output field | Meaning | |---|---| | `paperId`, `corpusId`, `url` | Semantic Scholar's own identifiers and the paper's S2 page | | `doi`, `arxivId`, `pubmedId`, `pubmedCentralId`, `dblpId` | Cross-reference IDs, join straight onto Crossref, arXiv, PubMed or DBLP | | `title`, `abstract` | Title and full abstract | | `tldr` | One-sentence AI summary generated by Semantic Scholar's TLDR model | | `year`, `publicationDate`, `venue` | Publication year, exact date where known, and the venue or conference | | `journalName`, `journalVolume`, `journalPages` | Journal placement where the record has it | | `authors`, `authorIds` | Author names in order, plus their S2 author IDs | | `citationCount`, `influentialCitationCount`, `referenceCount` | How often it is cited, how often that citation was *load-bearing*, and how many works it cites | | `fieldsOfStudy`, `publicationTypes` | Subject classification and record type (JournalArticle, Conference, Review…) | | `isOpenAccess`, `openAccessPdfUrl`, `openAccessStatus`, `openAccessLicense` | Whether a free PDF exists, where, and under which licence | | `citations` | Papers citing this one, only when `includeCitations` is on | | `scrapedAt` | Run timestamp | A `RUN_SUMMARY` record in the key-value store holds per-run counts, the filters used, and any query or paper ID that failed. ### Common use cases - **Systematic reviews.** Sweep a topic, filter by year and citation floor, and export straight to CSV. - **RAG and knowledge bases.** `title` + `abstract` + `tldr` + `fieldsOfStudy` is a ready-made chunk with metadata attached. - **Bibliometrics.** Track how `citationCount` and `influentialCitationCount` move for a paper set over months. - **Full-text collection.** Turn on `openAccessOnly` and you get a list of directly downloadable PDFs. - **ID reconciliation.** Feed in DOIs from another system and get back arXiv, PubMed and DBLP identifiers for the same works. ### Quick start One topic, the 100 most relevant papers: ```json { "searchQueries": ["retrieval augmented generation"] } ``` Recent, well-cited, open-access work across three topics, 200 papers in total: ```json { "searchQueries": [ "retrieval augmented generation", "vector database embedding search", "mixture of experts language model" ], "year": "2022-2026", "minCitationCount": 25, "openAccessOnly": true, "maxResults": 200, "apiKey": "YOUR_S2_KEY" } ``` Enrich a list of DOIs you already have: ```json { "paperIds": ["10.18653/v1/N19-1423", "arXiv:1706.03762", "PMID:19872477"], "fields": ["abstract", "tldr", "authors"] } ``` ### Input | Field | Type | Default | Notes | |---|---|---|---| | `searchQueries` | array | `[]` | Free-text searches, one per entry. No field syntax, no boolean operators, plain keywords, like the semanticscholar.org search box. Each query paginates independently. | | `paperIds` | array | `[]` | Specific papers. S2 hash, bare or prefixed DOI, `arXiv:1706.03762`, `PMID:19872477`, or a doi.org / arxiv.org / semanticscholar.org URL. | | `fields` | array | all seven | Optional fields per record: `abstract`, `tldr`, `authors`, `openAccessPdf`, `fieldsOfStudy`, `publicationTypes`, `journal`. Core metadata is always fetched. | | `includeCitations` | boolean | `false` | Add the list of citing papers to each record. Large payload, off by default. | | `year` | string | - | `2021`, `2015-2020`, `2020-` or `-2015`. Search only. | | `minCitationCount` | integer | `0` | Drop search hits below this citation count. | | `openAccessOnly` | boolean | `false` | Keep only papers with a downloadable PDF. | | `maxResults` | integer | `100` | Papers saved by the **whole run**, searches and lookups together. Shared out between the requests, not given to each. Ceiling 999. | | `apiKey` | string | - | Free key from semanticscholar.org/product/api. Strongly recommended. | At least one of `searchQueries` or `paperIds` is required, there is no meaningful "everything" mode. The two combine freely: papers you name are fetched first, the searches then share whatever is left of `maxResults`, and results are de-duplicated by `paperId`, so a paper found twice is saved (and charged) once. ### Output example A real record, fetched by DOI: ```json { "paperId": "df2b0e26d0599ce3e70df8a9da02e51594e0e992", "corpusId": 52967399, "doi": "10.18653/v1/N19-1423", "arxivId": "1810.04805", "pubmedId": null, "pubmedCentralId": null, "dblpId": "journals/corr/abs-1810-04805", "title": "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", "abstract": "We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. …", "tldr": "A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.", "year": 2019, "publicationDate": null, "venue": "North American Chapter of the Association for Computational Linguistics", "journalName": null, "journalVolume": null, "journalPages": "4171-4186", "authors": ["Jacob Devlin", "Ming-Wei Chang", "Kenton Lee", "Kristina Toutanova"], "authorIds": ["39172707", "1744179", "2544107", "3259253"], "citationCount": 118946, "influentialCitationCount": 22739, "referenceCount": 63, "fieldsOfStudy": ["Computer Science"], "publicationTypes": ["JournalArticle", "Conference"], "isOpenAccess": true, "openAccessPdfUrl": "https://doi.org/10.18653/v1/n19-1423", "openAccessStatus": "GOLD", "openAccessLicense": "CCBY", "citations": null, "url": "https://www.semanticscholar.org/paper/df2b0e26d0599ce3e70df8a9da02e51594e0e992", "scrapedAt": "2026-08-06T12:41:07.882Z" } ``` `publicationDate` is `null` for a good share of conference papers, Semantic Scholar only has `year` for them. `tldr` is present for most computer-science papers and thinner elsewhere, because the TLDR model was trained on that corpus. ### Finding a paper ID You rarely need one, searching is usually easier, but every form below works in `paperIds`: | What you have | What to paste | |---|---| | A DOI | `10.18653/v1/N19-1423`, or the same thing prefixed, `DOI:10.18653/v1/N19-1423` | | An arXiv paper | `arXiv:1706.03762` or `https://arxiv.org/abs/1706.03762` | | A PubMed record | `PMID:19872477` or the pubmed.ncbi.nlm.nih.gov URL | | A Semantic Scholar page | The whole URL, or just the 40-character hash at the end | The Actor strips URLs down to the identifier itself, so pasting links straight out of a browser is fine. ### API example ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~semantic-scholar-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "searchQueries": ["retrieval augmented generation"], "year": "2023-2026", "minCitationCount": 20, "maxResults": 50 }' ``` ### Limits and behaviour - **`maxResults` bounds the run, not the query.** It is the total number of papers the run may save, so it is also the most you can be charged for. The remaining allowance is divided among the requests still to come, so a query that returns little leaves more for the ones behind it. Requests the cap leaves no room for are never issued, and are listed in `RUN_SUMMARY.requestsSkipped` rather than dropped quietly. A value outside 1-999 is refused before the first request, never read as "no limit". - **Relevance search reaches 999 papers per query and no further.** That is why 999 is also the ceiling for `maxResults`. To go deeper, split the topic into narrower queries or slice it by `year`. - **Throttling is the constraint when you run without a key.** Semantic Scholar limits how fast it will answer, and at a busy moment that can mean a long wait or no answer at all. The Actor paces itself, retries with a growing gap, and gives up on a bounded budget rather than retrying forever. A request that never gets through is named in `RUN_SUMMARY.failures` with the reason and with however many papers it had already saved, and `requestsThrottled` counts them; a thin run therefore always says why it is thin. A free key removes most of this, get one before running anything large. - **Retries distinguish fatal from transient.** A malformed request and an unknown paper fail immediately, because retrying cannot help. Throttling and server errors back off, honouring a wait hint when one is given. - **A failing query never aborts the run.** It is logged and recorded in `RUN_SUMMARY.failures`; the remaining queries continue. The run only errors out if *every* request fails. - **Only requested fields are fetched.** The Actor asks for exactly the fields it maps, which keeps responses small; an unrecognised entry in `fields` is warned about and dropped rather than sent. - **Public data only.** No credentials beyond the optional key, no personal data, no access-control bypass. ### FAQ **Do I need a proxy?** No. Proxy configuration is not required to run this Actor. **Do I need a Semantic Scholar account?** Not to run it. The free API key is optional, and worth having for large runs: without one a busy moment can leave a query in `RUN_SUMMARY.failures`. **What happens if a query fails?** It is named in `RUN_SUMMARY.failures`, with the reason and the number of papers it had already saved, and the run continues with the rest. A search that simply matched nothing is listed separately under `queriesWithNoMatches`, so an empty result is never mistaken for a lost one. The run only errors out when every request failed. **Can I schedule it?** Yes. Citation counts move constantly, so re-running a fixed `paperIds` list on a schedule and diffing `citationCount` is a common pattern. **What is an "influential" citation?** Semantic Scholar's own classifier flags citations where the cited work shapes the citing paper, rather than appearing once in a related-work list. It is usually 5-20% of the total. **Does it fetch full text?** No, metadata plus the abstract. `openAccessPdfUrl` is the direct link when a free PDF exists. **Why is `tldr` empty for my papers?** The TLDR model covers computer science best and thins out in other fields. Nothing is inferred when the API has no TLDR. **Can I integrate it with something else?** Yes, Apify API, client libraries, webhooks, scheduled runs, dataset exports (JSON/CSV/Excel) or MCP. Output is structured JSON. # Actor input Schema ## `searchQueries` (type: `array`): Free-text searches, one per entry. the same thing you would type into semanticscholar.org. Plain keywords work best; there is no field syntax and no boolean operators. Each query is paginated separately and results are merged. ## `paperIds` (type: `array`): Fetch specific papers instead of, or alongside, searching. Accepts a 40-character Semantic Scholar ID, a bare or prefixed DOI (10.18653/v1/N19-1423), an arXiv ID (arXiv:1706.03762), PMID:19872477, or a full doi.org / arxiv.org / semanticscholar.org URL. An identifier Semantic Scholar does not know is named in RUN\_SUMMARY.failures, never returned as an empty row. Named papers are fetched before any search runs. ## `fields` (type: `array`): Optional fields added to every record. Core metadata (IDs, title, year, venue, citation counts) is always included. Unticking the heavy ones (abstract, TLDR, authors) makes responses smaller, which matters most on large runs. ## `includeCitations` (type: `boolean`): Add a 'citations' array to each record listing the papers that cite it (ID, title, year, citation count). Semantic Scholar returns at most the first 1,000 citing papers per record and the payload grows fast, so leave this off unless you are building a citation graph. ## `year` (type: `string`): Restrict searches by publication year: a single year ('2021'), a closed range ('2015-2020'), or an open one ('2020-' for 2020 onwards, '-2015' for everything up to 2015). Ignored for direct paper lookups. ## `minCitationCount` (type: `integer`): Drop search hits below this citation count. Useful for cutting a broad query down to work that has actually been picked up. 0 keeps everything. ## `openAccessOnly` (type: `boolean`): Keep only papers Semantic Scholar has a downloadable PDF for. Turn this on when the point of the run is to collect full texts rather than metadata. ## `maxResults` (type: `integer`): Total papers this run may save, across every search query and paper lookup together, and therefore the most you can be charged for. The cap is shared out between the requests, not given to each one: three queries at 60 return 60 rows in total, not 180. Relevance search reaches 999 papers per query, so 999 is the ceiling here too. Values outside 1-999 are refused rather than treated as unlimited. ## `apiKey` (type: `string`): Optional, and worth setting for anything large. A key gives the run its own allowance at Semantic Scholar, so a long run finishes sooner and is far less likely to leave a query in RUN\_SUMMARY.failures. Free from semanticscholar.org/product/api. ## Actor input object example ```json { "searchQueries": [ "graph neural network", "protein folding transformer" ], "paperIds": [ "10.18653/v1/N19-1423", "arXiv:1706.03762" ], "fields": [ "abstract", "tldr", "authors" ], "includeCitations": false, "year": "2020-2026", "minCitationCount": 0, "openAccessOnly": false, "maxResults": 100, "apiKey": "your-semantic-scholar-api-key" } ``` # Actor output Schema ## `items` (type: `string`): Every record the run produced. ## `runsummary` (type: `string`): The RUN\_SUMMARY record from the run's key-value store. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "searchQueries": [ "retrieval augmented generation" ] }; // Run the Actor and wait for it to finish const run = await client.actor("arman-bd/semantic-scholar-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "searchQueries": ["retrieval augmented generation"] } # Run the Actor and wait for it to finish run = client.actor("arman-bd/semantic-scholar-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "searchQueries": [ "retrieval augmented generation" ] }' | apify call arman-bd/semantic-scholar-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,arman-bd/semantic-scholar-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/bKIZBLV7Uo1l1avnG/builds/d8XpxeRP5WK1MuHzg/openapi.json