Semantic Scholar Scraper

Search and extract academic paper data from Semantic Scholar. Find research papers, analyze citations, and track references. No API key needed.

📖 What does Semantic Scholar Scraper do?

Semantic Scholar Scraper extracts academic research data from Semantic Scholar's database of 200M+ papers. Four modes:

🔹 Search — Find papers by keyword, filter by year, field of study, and citation count 🔹 Details — Get full metadata for specific papers by ID or DOI 🔹 Citations — List all papers that cite a given paper (who cited this?) 🔹 References — List all papers referenced by a given paper (bibliography)

❓ Why use Semantic Scholar Scraper?

🔹 No API key needed — Uses the public Semantic Scholar Academic Graph API 🔹 200M+ papers — Access one of the largest academic paper databases 🔹 Citation analysis — Track who cites a paper and build citation graphs 🔹 Open access detection — Find papers with free PDF access 🔹 Rich metadata — Authors, abstracts, venues, DOIs, ArXiv IDs, fields of study 🔹 Influential citations — Distinguish routine citations from influential ones

💡 Use cases

🔹 Literature reviews — Find all relevant papers on a topic with citation counts 🔹 Research tracking — Monitor new publications in your field 🔹 Citation analysis — Build citation networks and find influential papers 🔹 Academic SEO — Track citation impact of your publications 🔹 Competitive research — Monitor competitor institutions' publications 🔹 Dataset building — Create structured datasets of academic literature for ML/NLP

📊 Sample output

Paper data

Field	Example
title	Attention Is All You Need
year	2017
citationCount	120000
authors	Ashish Vaswani, Noam Shazeer, ...
venue	NeurIPS
doi	10.48550/arXiv.1706.03762
fieldsOfStudy	Computer Science
isOpenAccess	true

💰 Pricing

Event	Price
Start (per run)	$0.005
Paper scraped	$0.001

Free plan estimate: ~200 papers per month on the Apify Free plan.

🔢 How to search academic papers

1. Go to the Semantic Scholar Scraper page on Apify
2. Select mode (search, details, citations, or references)
3. Enter keywords or paper IDs
4. Set filters (year, field of study, minimum citations)
5. Click "Start" and download results as JSON, CSV, or Excel

📥 Input parameters

Parameter	Type	Description
mode	string	search, details, citations, or references
searchTerms	string[]	Keywords to search (search mode)
paperIds	string[]	Paper IDs, DOIs, or ArXiv IDs (details/citations/references mode)
year	string	Year filter (e.g. "2023", "2020-2024")
fieldsOfStudy	string[]	Research fields to filter by
openAccessOnly	boolean	Only papers with free PDF (default: false)
minCitations	number	Minimum citation count filter
maxResults	number	Max papers per query (default: 50)

📤 Output fields

type, paperId, title, year, citationCount, authors, authorIds, abstract, venue, publicationDate, doi, arxivId, url, pdfUrl, pdfLicense, fieldsOfStudy, isOpenAccess, influentialCitationCount, referenceCount, searchTerm, scrapedAt

💡 Tips

🔹 Paper IDs — You can use Semantic Scholar IDs (40-char hex), DOIs (prefix with DOI:), ArXiv IDs (prefix with ARXIV:), or Corpus IDs (prefix with CorpusId:). 🔹 Year ranges — Use 2020-2024 for a range, or 2023- for 2023 onwards. 🔹 Fields of study — Common values: Computer Science, Medicine, Physics, Biology, Chemistry, Mathematics, Economics, Psychology, Engineering. 🔹 Influential citations — The influentialCitationCount field counts citations that meaningfully build on the work (not just routine mentions). 🔹 Rate limits — Semantic Scholar allows ~1 request/second without an API key. For higher throughput, request a free key at semanticscholar.org/product/api.

🔗 Integrations

Export paper data to Google Sheets, Slack, Zapier, Make, or any webhook. Connect via the Apify API for automated research monitoring. Schedule weekly runs to track new publications.

💻 API usage

Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('automation-lab/semantic-scholar-scraper').call({
    mode: 'search',
    searchTerms: ['transformer neural network'],
    year: '2023-',
    minCitations: 100,
    maxResults: 50,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

from apify_client import ApifyClient

client = ApifyClient('YOUR_TOKEN')
run = client.actor('automation-lab/semantic-scholar-scraper').call(run_input={
    'mode': 'citations',
    'paperIds': ['649def34f8be52c8b66281af98ae884c09aef38b'],
    'maxResults': 100,
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)

cURL

curl "https://api.apify.com/v2/acts/automation-lab~semantic-scholar-scraper/runs" \
  -X POST -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{"mode": "search", "searchTerms": ["CRISPR gene editing"], "maxResults": 50}'

⚖️ Legality

Semantic Scholar Scraper accesses publicly available data through the official Semantic Scholar Academic Graph API. This API is provided by the Allen Institute for AI (AI2) and is designed for programmatic access to academic paper metadata. All data is derived from public academic publications.

❓ FAQ

Q: Do I need an API key? A: No. The Semantic Scholar API works without authentication, with a rate limit of ~1 request/second. For higher throughput, you can get a free API key from Semantic Scholar.

Q: How many papers are in the database? A: Over 200 million papers from all fields of science, indexed from major publishers, ArXiv, PubMed, and other sources.

Q: Can I search by author? A: The search mode uses keyword matching on titles and abstracts. For author-specific searches, find a paper by that author first, then use the details or citations mode.

Q: What's the difference between citations and references? A: Citations are papers that cite the target paper (who cited this?). References are papers that the target paper cites (its bibliography).

Q: Does this include full paper text? A: No. The API provides metadata (title, abstract, authors, etc.) and the pdfUrl field links to open-access PDFs when available.

🔗 Related scrapers

🔹 ArXiv Scraper — Search and extract preprints from ArXiv 🔹 CrossRef Scraper — Extract DOI metadata and citation data 🔹 OpenAlex Scraper — Academic paper data from OpenAlex