Semantic Scholar Scraper — Papers, Authors & Citations

Scrape academic papers, authors, and citation data from the Semantic Scholar Academic Graph API — a free, open-access API covering 200M+ papers across all scientific domains. No API key required.

Features

• 🔍 Paper Search — Search 200M+ papers by keyword query with pagination
• 📄 Paper Details — Get full paper metadata including references and citing papers
• 👤 Author Search — Find authors by name across the entire academic graph
• 📚 Author Papers — Retrieve an author's complete publication list with citation counts
• 📊 Flexible Fields — Choose which fields to return (title, abstract, authors, year, venue, openAccessPdf, and more)
• 🚀 Rate-Limit Aware — Automatic retry with backoff on HTTP 429, pagination up to 1000 results
• 💡 No API Key Needed — Works with the free tier (~100 requests per 5 minutes)

Use Cases

• Literature reviews — Gather all papers on a topic with abstracts and citation counts
• Citation analysis — Find the most-cited papers in a field or by a specific author
• Author profiling — Get an author's full publication list with metrics
• Open access discovery — Find open-access PDF URLs for papers
• Research monitoring — Track new papers by query or author on a schedule
• Academic data pipelines — Feed structured paper data into downstream analytics

Input

Field	Type	Required	Default	Description
mode	select	✅	search-papers	One of: search-papers, paper-details, author-search, author-papers
query	string	❌	"attention is all you need"	Search query (for search-papers and author-search)
paperId	string	❌	—	Semantic Scholar paper ID (for paper-details)
authorId	string	❌	—	Semantic Scholar author ID (for author-papers)
fields	string	❌	title,abstract,authors,year,citationCount,venue,openAccessPdf,url	Comma-separated fields to return
limit	int	❌	50	Max results (1–1000, paginated in batches of 100)
offset	int	❌	0	Number of results to skip

Output

Each dataset item is a JSON object. Fields depend on the fields input parameter and the mode.

Example: Paper Search

Input:

{
    "mode": "search-papers",
    "query": "attention is all you need",
    "limit": 3
}

Output (dataset items):

[
    {
        "paperId": "204e3073870fae3d05bcbc2f6a8e263d9b72e776",
        "title": "Attention Is All You Need",
        "abstract": "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks...",
        "authors": [
            { "authorId": "1741101", "name": "Ashish Vaswani" },
            { "authorId": "1726699", "name": "Noam Shazeer" }
        ],
        "year": 2017,
        "citationCount": 138523,
        "venue": "Advances in Neural Information Processing Systems",
        "openAccessPdf": { "url": "https://..." },
        "url": "https://www.semanticscholar.org/paper/204e3073870fae3d05bcbc2f6a8e263d9b72e776",
        "_mode": "search-papers"
    }
]

Example: Paper Details

Input:

{
    "mode": "paper-details",
    "paperId": "204e3073870fae3d05bcbc2f6a8e263d9b72e776",
    "fields": "title,abstract,authors,year,citationCount,references.title,citations.title"
}

Example: Author Search

Input:

{
    "mode": "author-search",
    "query": "Yann LeCun",
    "limit": 5
}

Example: Author Papers

Input:

{
    "mode": "author-papers",
    "authorId": "1726699",
    "limit": 10
}

How It Works

1. Reads input from the Apify input schema (mode, query, paperId, authorId, fields, limit, offset)
2. Constructs requests to the Semantic Scholar Academic Graph API (https://api.semanticscholar.org/graph/v1/)
3. Paginates results in batches of 100 (API max per page) up to the limit
4. Retries on HTTP 429 (rate limit) with exponential backoff
5. Pushes all results to the Apify dataset with a _mode tag

API Reference

This actor uses the Semantic Scholar Academic Graph API:

• GET /paper/search?query=<q>&limit=<n>&fields=<f> — Paper search
• GET /paper/{paperId}?fields=<f> — Paper details
• GET /author/search?query=<q>&limit=<n> — Author search
• GET /author/{authorId}/papers?fields=<f>&limit=<n> — Author's papers

Rate limit: ~100 requests per 5 minutes without an API key (sufficient for most use cases).

Pricing

$0.50 per 1,000 results — pay only for what you get.

At the default limit=50, a single run costs ~$0.025. The marginal cost per request is negligible (the API is free); the fee covers infrastructure and maintenance.

Resources

• Semantic Scholar API Documentation
• Semantic Scholar Website
• Apify Platform Documentation
• Apify Python SDK
• Integration with Make, Zapier, Google Drive, and more