🔬 ScienceDirect Search Scraper

🎯 Turn any ScienceDirect search into a clean, structured dataset of academic articles — in seconds, at scale.

This scraper collects research articles and papers from the ScienceDirect.com (Elsevier) search results. For every result you get the title, DOI, authors, journal, publication date, open-access status, citation details and PDF link — ready for literature reviews, meta-analyses, bibliometrics and research databases.

🚀 What Does It Do?

This scraper automatically searches ScienceDirect and collects structured, ready-to-use data for every matching article in the search results. No manual browsing, copy-pasting or page-flipping — just set your filters and hit Start.

💡 Two ways to run it:

1. 🔍 Discovery Mode — Enter keywords plus optional filters (author, journal, year, article type, open access) and the scraper builds the search, pages through every result and collects the articles.
2. 📋 Direct URL Mode — Already have a ScienceDirect search you like? Paste its search URL and the scraper pages through every matching result for you.

👥 Who Is This For?

🏢 Use Case	💬 How It Helps
🎓 Researchers & PhD students	Build a complete corpus for a systematic literature review without hours of manual searching
📊 Bibliometrics & scientometrics teams	Export thousands of citation records for trend, co-author and impact analysis
🤖 AI / NLP engineers	Assemble large metadata datasets to train and evaluate models
🏥 Pharma & R&D analysts	Track new publications in a field, journal or by a competitor's authors
📚 Librarians & info specialists	Deliver curated reading lists and current-awareness alerts to faculty
✍️ Science writers & journalists	Find the latest open-access papers on any topic, fast

✨ Features

• 🔍 Keyword & advanced search — Search by topic, title, author, journal, volume, issue and page
• 📅 Year & date filtering — One year, a list of years, or an inclusive range
• 🏷️ Article-type filters — Research articles, reviews, book chapters, data articles, conference abstracts and more
• 🔓 Open-access filter — Restrict to freely readable open-access and open-archive articles
• 🔃 Sort control — Relevance or newest-first
• 📚 Rich search metadata — Title, DOI, authors, journal, ISSN, dates, volume/issue, pages and article type for every result
• 🔗 DOI & PDF links — Every record includes its DOI and a direct PDF link where available
• 🧹 Smart Deduplication — Never get the same article twice, even across many result pages
• ⚡ Fast & Scalable — Collect anywhere from a handful to thousands of articles per run
• 📤 Export Anywhere — Download results as JSON, CSV, Excel, or push to Google Sheets, Zapier, Make, or your CRM

🎛️ Filters & Options

Option	What It Does
🔎 Search query	Main query across the search index
🏷️ Title contains	Restrict to articles whose title contains your words
👤 Author(s)	Filter by author name
📕 Publication title	Restrict to a specific journal or book
🔢 Volume / Issue / Page	Pinpoint a specific volume, issue or page
📅 Year(s)	Single year, comma list, or range like 2020-2025
🗂️ Article types	Limit to research articles, reviews, data articles, and more
🔓 Open access only	Keep only freely readable articles
🔃 Sort by	Relevance or date (newest first)
🔗 Start URLs	Optionally provide specific ScienceDirect search URLs
🔢 Max items	Control how many articles to extract per run
📑 Max pages per search	Cap how many result pages to page through per search
⚙️ Max concurrency	Control how many pages are processed in parallel
🌐 Proxy configuration	Route requests through Apify's proxy service

📦 What You Get (Output Fields)

Every article includes:

Bibliographic

Field	Example
title	A Machine Learning Algorithm to Estimate the Probability...
doi	10.1016/j.jhsa.2025.01.021
authors	["Anne Eva J. Bulstra"]
journalName	The Journal of Hand Surgery
journalUrl	https://www.sciencedirect.com/science/journal/03635023
issn	03635023
volumeIssue	Volume 50, Issue 6
pages	711-720
publishedDate	June 2025
onlineDate	2025-06-03T00:00:00.000Z
articleType	Research article
contentType	JL
language	en

Access & Links

Field	Example
isOpenAccess	false
pdfUrl	https://www.sciencedirect.com/science/article/pii/S036350232500036X/pdfft?pid=...
articleUrl	https://www.sciencedirect.com/science/article/pii/S036350232500036X
pii	S036350232500036X
searchUrl	https://www.sciencedirect.com/search?qs=machine+learning&show=100
scrapedAt	2026-07-01T08:10:34.590Z

📊 Example Output

{
    "doi": "10.1016/j.jhsa.2025.01.021",
    "title": "A Machine Learning Algorithm to Estimate the Probability of a True Scaphoid Fracture After Wrist Trauma",
    "articleUrl": "https://www.sciencedirect.com/science/article/pii/S036350232500036X",
    "pii": "S036350232500036X",
    "articleType": "Research article",
    "contentType": "JL",
    "journalName": "The Journal of Hand Surgery",
    "journalUrl": "https://www.sciencedirect.com/science/journal/03635023",
    "issn": "03635023",
    "publishedDate": "June 2025",
    "onlineDate": "2025-06-03T00:00:00.000Z",
    "volumeIssue": "Volume 50, Issue 6",
    "pages": "711-720",
    "authors": ["Anne Eva J. Bulstra"],
    "language": "en",
    "isOpenAccess": false,
    "pdfUrl": "https://www.sciencedirect.com/science/article/pii/S036350232500036X/pdfft?pid=1-s2.0-S036350232500036X-main.pdf",
    "searchUrl": "https://www.sciencedirect.com/search?qs=machine+learning&show=100",
    "scrapedAt": "2026-07-01T08:10:34.590Z"
}

📋 Dataset Views

The Apify Console gives you 3 ready-made table views to quickly browse your results:

View	What It Shows
📊 Overview	Title, authors, journal, date, open-access flag, DOI and link
📚 Bibliographic	Full citation metadata — journal, ISSN, volume/issue, pages, dates
📋 Full Details	Every single field — the complete dataset

❓ FAQ

🤔 Can I scrape a search I already built on the website? Yes — just paste the ScienceDirect search URL into Start URLs and the scraper will page through every result for you.

🤔 What data do I get for each article? You get rich search-listing metadata — title, DOI, authors, journal, ISSN, volume/issue, pages, publication and online dates, article type, language, open-access status, and the article and PDF links. Detail-page-only fields such as abstract, keywords and author affiliations are not collected.

🤔 Can it collect open-access PDFs? The scraper returns the direct PDF link for every article where one is available, and you can filter to open-access articles only.

🤔 How many articles can I collect? From a few up to thousands per run — set the Max items limit to whatever you need.

🤔 Can I export the data? Yes — JSON, CSV, Excel, XML, HTML, RSS. You can also push data directly to Google Sheets, Zapier, Make, or any webhook/API endpoint.

🤔 How often should I run this? For fresh data, run daily or weekly. You can schedule automatic runs on Apify with just a few clicks.

🤔 Does it work with proxies? Yes — residential proxies are recommended for the best results. The scraper works with Apify's built-in proxy service.

🛠️ Need Custom Filters or Features?

I'm happy to customize this scraper for your specific needs! 🤝

Whether you need:

• 🎯 Additional filters (subject area, affiliation country, citation count, specific journals)
• 📊 Extra data fields or custom output formats (RIS / BibTeX export, reference lists)
• 🔄 Integration with your CRM, Google Sheets, reference manager or database
• ⏰ Scheduled scraping with automatic deduplication and current-awareness alerts
• 🌐 Scraping from other academic platforms alongside ScienceDirect

👉 Don't hesitate to reach out via private message — I respond quickly and I'm always open to building exactly what you need. No request is too small or too specific!

⚖️ Legal & Ethical Use

This scraper collects only publicly available information from ScienceDirect search results. It does not access paywalled full text, bypass authentication, or use any institutional credentials. Please use the data responsibly and in compliance with applicable laws and ScienceDirect's terms of service.