# Semantic Scholar Scraper - Low-costπŸ’²πŸ”₯πŸ“šπŸ€– (`delectable_incubator/semantic-scholar-scraper-low-cost`) Actor πŸ“šπŸ”Ž Extract research papers from Semantic Scholar using keywords, paper URLs, or author profiles. Collect paper titles, authors, publication years, abstracts, citations, venues, research fields, paper URLs, and metadata. Ideal for academic research, literature reviews & AI research monitoring. πŸš€ - **URL**: https://apify.com/delectable\_incubator/semantic-scholar-scraper-low-cost.md - **Developed by:** [Prime Scrape](https://apify.com/delectable_incubator) (community) - **Categories:** Jobs, Lead generation, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.00005 / actor start This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README

Semantic Scholar Papers Scraper

--- ## πŸ“šπŸ€– Semantic Scholar Papers Scraper | Bulk Research Papers Scraper | Apify Actor ### πŸš€ Extract Semantic Scholar Papers in Seconds (No Code) The **Semantic Scholar Papers Scraper (Apify Actor)** is a powerful, scalable, and SEO-optimized academic paper scraping tool built for researchers, universities, AI companies, startups, and data analysts. Extract research papers in bulk directly from Semantic Scholar search pages, author profiles, and publication listings. Collect structured academic metadata for literature reviews, citation analysis, AI datasets, bibliometric research, trend monitoring, and scientific intelligence. --- ## πŸ”₯ Why This Semantic Scholar Scraper? βœ” Best Semantic Scholar scraper on Apify βœ” Bulk URL scraping support (multi-URL mode πŸš€) βœ” Scrape search pages, author profiles & paper listings βœ” Automatic pagination βœ” High-speed extraction engine βœ” Clean structured JSON / CSV / Excel output βœ” Perfect for AI research datasets βœ” Ideal for universities, researchers & companies βœ” No coding required --- ## 🎯 What This Scraper Does This Apify Actor extracts real-time academic research papers directly from Semantic Scholar. ### πŸ“Œ Core Features βœ… Scrape Semantic Scholar search results βœ… Bulk URL scraping (SEO BOOST πŸš€) βœ… Supports multiple search URLs βœ… Supports author profile URLs βœ… Automatic pagination βœ… Respect maxItemsPerUrl limits βœ… Extract titles, abstracts, authors, citations βœ… Extract journal & conference information βœ… Extract PDF links when available βœ… Extract Open Access links βœ… Clean structured datasets βœ… Fast cloud execution --- ## ⚑ Input Configuration (Simple & Powerful) ### πŸ”₯ BULK URL MODE (SEO BOOST πŸš€) ```` { "urls": \[ "https://www.semanticscholar.org/search?q=artificial+intelligence", "https://www.semanticscholar.org/search?q=machine+learning", "https://www.semanticscholar.org/search?q=deep+learning", "https://www.semanticscholar.org/search?q=computer+vision" ], "maxItemsPerUrl": 100 } ``` --- ## πŸ“Š Extracted Paper Data (Structured Output) | Field | Description | |------|-------------| | paperId | Unique paper identifier | | title | Paper title | | abstract | Paper abstract | | authors | Authors list | | authorIds | Author identifiers | | year | Publication year | | publicationDate | Publication date | | venue | Journal or conference | | journalName | Journal name | | publicationTypes | Publication type | | fieldsOfStudy | Scientific fields | | citationCount | Citation count | | influentialCitationCount | Influential citations | | referenceCount | Number of references | | openAccessPdf | Open Access PDF | | pdfUrl | PDF download URL | | paperUrl | Paper page | | sourceUrl | Source search URL | | scrapeTimestamp | Extraction timestamp | --- ## πŸ’‘ Use Cases (High Demand SEO Keywords) This Semantic Scholar scraper is perfect for: πŸ“š Literature reviews πŸ€– AI research datasets πŸ“Š Citation analysis πŸ”¬ Scientific trend monitoring 🏫 University research πŸ“ˆ Bibliometric analysis 🧠 Machine Learning datasets πŸ“‘ Academic search automation 🌍 Research intelligence ⚑ Bulk paper extraction --- ## πŸš€ Key Features (Apify SEO Optimized) ⚑ Bulk URL scraping support πŸ“š Academic research extraction πŸ“„ Search pages & author profiles πŸ”Ž Automatic pagination πŸ“Š Structured metadata 🧠 Citation analytics πŸ“₯ PDF detection πŸ’Ύ Export-ready datasets βš™οΈ Cloud scalable architecture πŸ” Reliable retry system --- ## πŸ“€ Output Formats Supported βœ” JSON βœ” CSV βœ” Excel XLSX βœ” XML βœ” HTML --- ## πŸ“¦ Example Output ``` { "paperId": "530a059cb48477ad1e3d4f8f4b153274c8997332", "title": "Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI", "authors": \[ "Alejandro Barredo Arrieta", "Natalia DΓ­az RodrΓ­guez", "Francisco Herrera" ], "year": 2019, "venue": "Information Fusion", "journalName": "Information Fusion", "fieldsOfStudy": \[ "Computer Science", "Philosophy" ], "citationCount": 1450, "referenceCount": 312, "pdfUrl": "https://arxiv.org/pdf/1910.10045.pdf", "paperUrl": "https://www.semanticscholar.org/paper/530a059cb48477ad1e3d4f8f4b153274c8997332", "sourceUrl": "https://www.semanticscholar.org/search?q=artificial+intelligence" } ```` --- ## πŸ“Š Preconfigured Dataset Views ### πŸ”Ž Overview View Quick overview of collected papers. Fields included: β€’ Paper ID β€’ Title β€’ Authors β€’ Year β€’ Citation Count β€’ Source URL --- ### πŸ“š Detailed View Complete academic metadata. Fields included: β€’ Paper ID β€’ Title β€’ Authors β€’ Abstract β€’ Venue β€’ Journal β€’ Publication Date β€’ Citation Count β€’ Reference Count β€’ Fields of Study β€’ Publication Type β€’ Open Access PDF β€’ PDF URL β€’ Paper URL β€’ Source URL --- ### 🌐 By Source URL View Group papers by each search URL. Fields included: β€’ Source URL β€’ Paper ID β€’ Title β€’ Authors β€’ Year β€’ Citation Count β€’ Paper URL --- ## πŸ”₯ Why This is the BEST Semantic Scholar Scraper on Apify? βœ” Optimized for Apify Marketplace SEO βœ” Supports bulk URL scraping βœ” Extremely fast extraction βœ” Handles automatic pagination βœ” Clean academic metadata βœ” Enterprise-ready architecture βœ” Research-grade datasets βœ” Perfect for universities, AI companies & researchers --- ## πŸ’Έ Pricing This scraper runs on a **pay-per-result pricing model**. You only pay for successfully extracted records. πŸ’³ **Price:** **$2.99 / 1,000 results** --- ## ❓ FAQ (SEO BOOST SECTION) #### Can I scrape multiple URLs at once? Yes β€” Bulk URL mode is fully supported. #### Can I scrape author pages? Yes β€” author profile URLs are supported. #### Does it paginate automatically? Yes β€” pagination is handled automatically. #### Can I collect citation counts? Yes β€” citation statistics are extracted whenever available. #### Can I download PDF links? Yes β€” PDF and Open Access links are extracted when available. #### Is coding required? No β€” this is a 100% no-code Apify Actor. #### Can I export my data? Yes β€” JSON, CSV, Excel, XML and HTML are supported. --- ## ⚠️ Disclaimer This tool is an independent data extraction solution and is not affiliated with, endorsed by, or sponsored by Semantic Scholar. --- ## πŸ”— Related Actors (Academic Intelligence Suite) We are building a complete **PrimeScrape Academic Intelligence Suite**. More academic scraping tools are coming soon πŸš€ --- ## 🌍 PrimeScrape Ecosystem Built for large-scale data extraction, automation, AI datasets, and research intelligence. πŸ“š Academic paper extraction πŸ€– AI research datasets πŸ“ˆ Scientific analytics 🏫 University research πŸ”¬ Citation intelligence πŸ“Š Bibliometric analysis βš™οΈ Automation pipelines 🌍 Market & research intelligence --- ## πŸ“¬ Support ⭐⭐⭐⭐⭐ Leave a review if you enjoy this scraper. πŸ“© Contact us for enterprise solutions, custom academic scrapers, or large-scale research projects. # Actor input Schema ## `urls` (type: `array`): One or more Semantic Scholar search URLs. Examples: β€’ https://www.semanticscholar.org/search?q=web&sort=relevance β€’ https://www.semanticscholar.org/search?q=machine+learning&sort=relevance β€’ https://www.semanticscholar.org/search?q=deep+learning&sort=Relevance&fos[]=Computer+Science ## `maxItemsPerUrl` (type: `integer`): Maximum number of papers to collect per URL. Set to 0 for no limit. ## `proxy` (type: `object`): Optional. ## Actor input object example ```json { "urls": [ "https://www.semanticscholar.org/search?q=web&sort=relevance", "https://www.semanticscholar.org/search?q=DEEP+learning&sort=relevance" ], "maxItemsPerUrl": 30, "proxy": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ], "apifyProxyCountry": "US" } } ```` # Actor output Schema ## `overview` (type: `string`): No description ## `detailed` (type: `string`): No description ## `by_source` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "https://www.semanticscholar.org/search?q=web&sort=relevance", "https://www.semanticscholar.org/search?q=DEEP+learning&sort=relevance" ], "maxItemsPerUrl": 30, "proxy": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ], "apifyProxyCountry": "US" } }; // Run the Actor and wait for it to finish const run = await client.actor("delectable_incubator/semantic-scholar-scraper-low-cost").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": [ "https://www.semanticscholar.org/search?q=web&sort=relevance", "https://www.semanticscholar.org/search?q=DEEP+learning&sort=relevance", ], "maxItemsPerUrl": 30, "proxy": { "useApifyProxy": True, "apifyProxyGroups": ["RESIDENTIAL"], "apifyProxyCountry": "US", }, } # Run the Actor and wait for it to finish run = client.actor("delectable_incubator/semantic-scholar-scraper-low-cost").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "https://www.semanticscholar.org/search?q=web&sort=relevance", "https://www.semanticscholar.org/search?q=DEEP+learning&sort=relevance" ], "maxItemsPerUrl": 30, "proxy": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ], "apifyProxyCountry": "US" } }' | apify call delectable_incubator/semantic-scholar-scraper-low-cost --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=delectable_incubator/semantic-scholar-scraper-low-cost", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Semantic Scholar Scraper - Low-costπŸ’²πŸ”₯πŸ“šπŸ€–", "description": "πŸ“šπŸ”Ž Extract research papers from Semantic Scholar using keywords, paper URLs, or author profiles. Collect paper titles, authors, publication years, abstracts, citations, venues, research fields, paper URLs, and metadata. Ideal for academic research, literature reviews & AI research monitoring. πŸš€", "version": "0.0", "x-build-id": "wfRI70nhZWKEGunib" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/delectable_incubator~semantic-scholar-scraper-low-cost/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-delectable_incubator-semantic-scholar-scraper-low-cost", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/delectable_incubator~semantic-scholar-scraper-low-cost/runs": { "post": { "operationId": "runs-sync-delectable_incubator-semantic-scholar-scraper-low-cost", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/delectable_incubator~semantic-scholar-scraper-low-cost/run-sync": { "post": { "operationId": "run-sync-delectable_incubator-semantic-scholar-scraper-low-cost", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "urls" ], "properties": { "urls": { "title": "Search URLs", "type": "array", "description": "One or more Semantic Scholar search URLs.\n\nExamples:\nβ€’ https://www.semanticscholar.org/search?q=web&sort=relevance\nβ€’ https://www.semanticscholar.org/search?q=machine+learning&sort=relevance\nβ€’ https://www.semanticscholar.org/search?q=deep+learning&sort=Relevance&fos[]=Computer+Science", "items": { "type": "string" }, "default": [ "https://www.semanticscholar.org/search?q=web&sort=relevance", "https://www.semanticscholar.org/search?q=DEEP+learning&sort=relevance" ] }, "maxItemsPerUrl": { "title": "Max Items per URL", "type": "integer", "description": "Maximum number of papers to collect per URL. Set to 0 for no limit.", "default": 30 }, "proxy": { "title": "Proxy Configuration", "type": "object", "description": "Optional.", "default": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ], "apifyProxyCountry": "US" } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```