Semantic Scholar Papers Scraper

Here is one real result, with every field the actor returns:

{
  "paperId": "81a4fd3004df0eb05d6c1cef96ad33d5407820df",
  "title": "A Comprehensive Survey on Graph Neural Networks",
  "year": 2019,
  "venue": "IEEE Transactions on Neural Networks and Learning Systems",
  "citationCount": 11738,
  "authors": ["Zonghan Wu", "Shirui Pan", "Fengwen Chen", "Guodong Long", "Chengqi Zhang", "Philip S. Yu"],
  "doi": "10.1109/TNNLS.2020.2978386",
  "arxivId": "1901.00596",
  "abstract": "Deep learning has revolutionized many machine learning tasks in recent years, ranging from image classification and video processing to speech recognition and natural language understanding. The data in these tasks are typically represented in the Euclidean space. However, there is an increasing number of applications, where data are generated from non-Euclidean domains and are represented as graphs with complex relationships and interdependency between objects. [trimmed for length, real value continues] Finally, we propose potential research directions in this rapidly growing field.",
  "url": "https://www.semanticscholar.org/paper/81a4fd3004df0eb05d6c1cef96ad33d5407820df",
  "tldr": "This article provides a comprehensive overview of graph neural networks (GNNs) in data mining and machine learning fields and proposes a new taxonomy to divide the state-of-the-art GNNs into four categories.",
  "influentialCitationCount": 519,
  "referenceCount": 207,
  "isOpenAccess": true,
  "openAccessPdfUrl": "https://arxiv.org/pdf/1901.00596",
  "openAccessPdfLicense": null,
  "publicationDate": null,
  "publicationTypes": ["JournalArticle", "Review"],
  "fieldsOfStudy": ["Computer Science", "Mathematics", "Medicine"],
  "journalName": "IEEE Transactions on Neural Networks and Learning Systems",
  "journalVolume": "32",
  "journalPages": "4-24",
  "pubmedId": "32217482",
  "corpusId": "57375753",
  "authorIds": ["2109557884", "2585415", "31370754", "2062835", "48934799", "144019071"],
  "source": "Semantic Scholar",
  "observedAt": "2026-08-14T06:32:58.763Z",
  "aiSummary": "This paper provides a thorough overview of graph neural networks (GNNs), categorizing them into four types and discussing their applications in various fields. It also highlights challenges in graph data processing and suggests future research directions.",
  "aiKeywords": ["Graph Neural Networks", "Deep Learning", "Machine Learning", "Data Mining", "Non-Euclidean Data", "Taxonomy", "Recurrent GNNs", "Convolutional GNNs", "Graph Autoencoders", "Spatial-Temporal GNNs"],
  "aiField": "Machine Learning"
}

The most complete Semantic Scholar paper scraper available. It returns every field the Semantic Scholar search API exposes for each paper, from citation and reference counts to open-access PDF links, journal details and cross-index IDs, plus optional paid AI add-ons that summarize, tag and classify each paper.

📥 Input · 📤 Output · 💰 Pricing · ▶️ Examples

Table of contents

• What it does
• Quickstart
• Input reference
• Output reference
• Example output record
• Run via API and CLI
• Fetch results
• Billing and limits
• FAQ and troubleshooting

What it does

The actor searches the Semantic Scholar academic paper index by keyword, paginates through the matching papers, and writes one normalized record per paper to the run's dataset. Each record carries the bibliographic core (title, authors, year, venue, journal), impact metrics (citation count, influential citation count, reference count), the full abstract, open-access PDF link when available, and cross-index identifiers (DOI, arXiv, PubMed, corpus ID). Missing source values are returned as null and lists as empty arrays, never invented.

Three optional paid AI add-ons enrich each record when switched on: aiSummary (a plain-English summary of the abstract), aiKeywords (5 to 10 topical keywords), and aiField (a normalized academic discipline). AI add-ons run on paid Apify plans only and are billed per paper only when they produce usable output.

Quickstart

Open the actor, paste this into the input, and press Run. It returns the 10 most relevant papers for the query.

{
  "searchQuery": "large language models",
  "maxPapers": 10
}

To enable AI enrichment (paid plans), switch on the add-ons you want:

{
  "searchQuery": "graph neural networks",
  "maxPapers": 25,
  "withAiSummary": true,
  "withAiKeywords": true,
  "withAiField": true
}

searchQuery is required. Every other field is optional; the default maxPapers is 10.

Input reference

Field	Type	Required	Default	Description
searchQuery	string	yes	(none)	Keyword matched against paper title and abstract, for example large language models, gene therapy, quantum computing.
maxPapers	integer	no	10	Maximum number of papers to collect. Range 1 to 1000000. Free Apify plans are capped at 10 per run.
withAiSummary	boolean	no	false	Paid add-on. Generates a 1 to 2 sentence plain-English summary from each paper's abstract (skipped when no abstract). Requires a paid Apify plan. Billed only when a summary is produced.
withAiKeywords	boolean	no	false	Paid add-on. Extracts 5 to 10 topical keywords per paper from title, abstract and fields. Requires a paid Apify plan. Billed only when keywords are produced.
withAiField	boolean	no	false	Paid add-on. Classifies each paper's specific academic field or discipline. Requires a paid Apify plan. Billed only when a classification is produced.

Output reference

One dataset item per paper. Types: string, integer, boolean, string[], or null when the source value is absent.

Field	Type	Description
paperId	string	Semantic Scholar paper ID (unique per paper).
title	string	Paper title.
year	integer	Publication year, or null.
venue	string	Publication venue (conference or journal name).
citationCount	integer	Total number of citations recorded.
authors	string[]	Author display names in order.
doi	string	Digital Object Identifier, or null.
arxivId	string	arXiv identifier, or null.
abstract	string	Full paper abstract (capped at 1500 characters), or null.
url	string	Semantic Scholar page for the paper.
tldr	string	Semantic Scholar auto-generated one-line summary, or null.
influentialCitationCount	integer	Count of citations flagged as influential.
referenceCount	integer	Number of references in the paper.
isOpenAccess	boolean	true when an open-access version exists, else false or null.
openAccessPdfUrl	string	Direct link to an open-access PDF, or null.
openAccessPdfLicense	string	License of the open-access PDF, or null.
publicationDate	string	Full publication date (YYYY-MM-DD), or null.
publicationTypes	string[]	Publication type tags, for example JournalArticle, Review.
fieldsOfStudy	string[]	Fields of study assigned by Semantic Scholar.
journalName	string	Journal name, or null.
journalVolume	string	Journal volume, or null.
journalPages	string	Journal page range, or null.
pubmedId	string	PubMed identifier, or null.
corpusId	string	Semantic Scholar corpus ID.
authorIds	string[]	Semantic Scholar author IDs matching authors.
source	string	Always Semantic Scholar.
observedAt	string	ISO 8601 timestamp of when the record was collected.
aiSummary	string	Paid add-on. Plain-English abstract summary, or null when off or no abstract.
aiKeywords	string[]	Paid add-on. Topical keywords, or null when off.
aiField	string	Paid add-on. Normalized academic discipline, or null when off.

Example output record

Real record from a live run (input {"searchQuery": "graph neural networks", "maxPapers": 10, "withAiSummary": true, "withAiKeywords": true, "withAiField": true}):

{
  "paperId": "62ed9bf1d83c8db1f9cbf92ea2f57ea90ef683d9",
  "title": "How Powerful are Graph Neural Networks?",
  "year": 2018,
  "venue": "International Conference on Learning Representations",
  "citationCount": 10240,
  "authors": ["Keyulu Xu", "Weihua Hu", "J. Leskovec", "S. Jegelka"],
  "doi": null,
  "arxivId": "1810.00826",
  "abstract": "Graph Neural Networks (GNNs) are an effective framework for representation learning of graphs. GNNs follow a neighborhood aggregation scheme, where the representation vector of a node is computed by recursively aggregating and transforming representation vectors of its neighboring nodes. [trimmed for length, real value continues] We empirically validate our theoretical findings on a number of graph classification benchmarks, and demonstrate that our model achieves state-of-the-art performance.",
  "url": "https://www.semanticscholar.org/paper/62ed9bf1d83c8db1f9cbf92ea2f57ea90ef683d9",
  "tldr": "This work characterize the discriminative power of popular GNN variants, such as Graph Convolutional Networks and GraphSAGE, and show that they cannot learn to distinguish certain simple graph structures, and develops a simple architecture that is provably the most expressive among the class of GNNs.",
  "influentialCitationCount": 1806,
  "referenceCount": 44,
  "isOpenAccess": false,
  "openAccessPdfUrl": null,
  "openAccessPdfLicense": null,
  "publicationDate": "2018-10-01",
  "publicationTypes": ["JournalArticle"],
  "fieldsOfStudy": ["Computer Science", "Mathematics"],
  "journalName": "ArXiv",
  "journalVolume": "abs/1810.00826",
  "journalPages": null,
  "pubmedId": null,
  "corpusId": "52895589",
  "authorIds": ["3360632", "48594758", "1702139", "2594093"],
  "source": "Semantic Scholar",
  "observedAt": "2026-08-14T06:32:58.763Z",
  "aiSummary": "This study aims to analyze the expressive power of Graph Neural Networks (GNNs) in capturing different graph structures. The findings reveal that while many GNN variants excel in performance, they have limitations in distinguishing certain simple graph structures, and a new architecture developed in the study proves to be the most expressive among GNNs, achieving state-of-the-art results.",
  "aiKeywords": ["Graph Neural Networks", "representation learning", "neighborhood aggregation", "expressive power", "discriminative power", "Graph Convolutional Networks", "GraphSAGE", "graph classification", "theoretical framework", "Weisfeiler-Lehman"],
  "aiField": "Machine Learning"
}

Run via API and CLI

Start a run and wait for it to finish, then read the dataset. Replace <TOKEN> with your Apify API token.

Run synchronously and get dataset items in one call:

curl -X POST "https://api.apify.com/v2/acts/scrapers_lat~semantic-scholar-scraper/run-sync-get-dataset-items?token=<TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"searchQuery":"large language models","maxPapers":25}'

Start a run asynchronously:

curl -X POST "https://api.apify.com/v2/acts/scrapers_lat~semantic-scholar-scraper/runs?token=<TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"searchQuery":"gene therapy","maxPapers":100,"withAiKeywords":true}'

Apify CLI:

apify call scrapers_lat/semantic-scholar-scraper \
  --input '{"searchQuery":"quantum computing","maxPapers":10}'

Fetch results

Every run writes to a dataset. Fetch items as JSON, CSV, or Excel by changing format:

# JSON
curl "https://api.apify.com/v2/datasets/<DATASET_ID>/items?token=<TOKEN>&clean=true&format=json"

# CSV
curl "https://api.apify.com/v2/datasets/<DATASET_ID>/items?token=<TOKEN>&clean=true&format=csv"

# Paginate large datasets
curl "https://api.apify.com/v2/datasets/<DATASET_ID>/items?token=<TOKEN>&offset=1000&limit=1000"

<DATASET_ID> is returned as defaultDatasetId in the run object. Use offset and limit to page through large result sets. clean=true drops empty and internal fields.

Billing and limits

• Pay per result. You are charged per paper returned (result event). See the pricing tab for the current per-result price.
• AI add-ons are billed separately. ai_summary, ai_keywords, and ai_field each charge per paper only when they produce usable output, and only on paid Apify plans.
• No charge on failure. If a run errors, the actor writes a single item with a populated error field and does not charge for it. Empty runs cost nothing.
• Spend cap respected. Set maxTotalChargeUsd on the run; once reached, the actor stops emitting and charging further billable results.
• Free Apify plans are capped at 10 papers per run and cannot use the AI add-ons. Upgrade for higher maxPapers and AI enrichment.
• Source rate limits. The public Semantic Scholar API is shared and rate-limited; the actor retries within a wall-clock budget so runs finish reliably.

FAQ and troubleshooting

A run returned 0 papers. Why? The keyword matched nothing, or the source was rate-limiting for the whole run. Try a broader keyword and rerun. Zero-result runs are not charged.

Why is doi or openAccessPdfUrl null? Not every paper has a registered DOI or an open-access PDF. Missing source values are returned as null, never invented.

How do I get the AI summary, keywords, and field? Switch on withAiSummary, withAiKeywords, and withAiField. They require a paid Apify plan and are billed per paper only when they return usable output.

Can I pull thousands of papers? Yes, raise maxPapers. Very large pulls take longer because the source API is rate-limited and the actor paginates politely.

Is this an official Semantic Scholar tool? No. This actor is independent and has no affiliation with Semantic Scholar or the Allen Institute for AI. It reads only publicly available data. Use it in accordance with the Semantic Scholar terms of service.

Related scrapers

• arXiv Papers Scraper: arXiv preprints by keyword, author or category.
• Crossref Scraper: scholarly metadata and DOIs from Crossref.
• ClinicalTrials Scraper: registered clinical trials worldwide.
• GBIF Species Scraper: biodiversity occurrence and taxonomy records.
• EU CORDIS Projects Scraper: EU-funded research projects and participants.

More scrapers at scrapers.lat

Built and maintained by scrapers.lat, where we publish scrapers for US and Latin American public platforms: company registries, government data, finance, e-commerce and more. Browse the catalog or request a custom scraper at scrapers.lat.

---

Independent tool, not affiliated with Semantic Scholar or the Allen Institute for AI. Accesses only publicly available data. Use in accordance with the Semantic Scholar terms of service.