📚 ScienceDirect Search Scraper

This actor allows you to scrape academic research articles from ScienceDirect.com by providing a search URL. It automatically handles pagination and extracts comprehensive article information including titles, authors, journals, publication dates, and access details.

Features

• URL-based Scraping: Extract articles by providing a complete ScienceDirect search URL
• Automatic Pagination: Automatically clicks through pages and loads more articles until reaching the end or max items limit
• Comprehensive Article Data: Extract detailed information about each article including:
  • DOI (Digital Object Identifier)
  • Article title and direct link
  • Article type (Research article, Review, etc.)
  • Open access status
  • Journal name
  • Publication date
  • Author list
  • PDF download link (if available)
• Smart Deduplication: Automatically removes duplicate articles based on article URLs

Input Parameters

Field	Type	Required	Description
searchUrl	string	Yes	Complete ScienceDirect search URL to scrape
maxItems	integer	No	Maximum number of articles to fetch
proxyConfiguration	object	No	Proxy settings for the actor

How to Get Search URL

1. Go to ScienceDirect.com
2. Use the search bar to enter keywords, or apply filters (year, journal, subject area, etc.)
3. Copy the complete URL from your browser's address bar
4. Paste it into the searchUrl field

Example search URLs:

https://www.sciencedirect.com/search?qs=RPG&years=2025&lastSelectedFacet=years
https://www.sciencedirect.com/search?qs=machine%20learning&date=2024
https://www.sciencedirect.com/search?qs=climate%20change&subjects=Environmental%20Science

Output

The output is a dataset of article objects, each containing:

• searchUrl: The original search URL used
• doi: Digital Object Identifier of the article
• title: Full title of the article
• articleUrl: Direct link to the article detail page
• articleType: Type of article (e.g., "Research article", "Review article")
• isOpenAccess: Boolean indicating if the article is open access
• journalName: Name of the journal where the article was published
• publishedDate: Publication date (e.g., "November 2025")
• authors: Array of author names
• pdfUrl: URL to download the PDF version (if available, null otherwise)
• scrapedAt: Timestamp of when the article was scraped

Example output:

[
  {
    "searchUrl": "https://www.sciencedirect.com/search?qs=RPG&years=2025&lastSelectedFacet=years",
    "doi": "10.1016/j.chiabu.2025.107729",
    "title": "A further look into the impact of the Regional Partnership Grant (RPG) program on adult and child outcomes: Evidence from Cohort Five",
    "articleUrl": "https://www.sciencedirect.com/science/article/pii/S0145213425004855",
    "articleType": "Research article",
    "isOpenAccess": false,
    "journalName": "Child Abuse & Neglect",
    "publishedDate": "November 2025",
    "authors": [
      "Yange Xue",
      "Ning Fu",
      "Tara Wildszewski"
    ],
    "pdfUrl": null,
    "scrapedAt": "2025-11-12T06:53:42.333Z"
  },
  ...
]

Use Cases

• Academic research and literature review
• Citation analysis and tracking
• Journal monitoring and analysis
• Research trend identification
• Author publication tracking
• Open access article discovery
• Academic database building
• Research collaboration discovery

Notes

• Results are automatically paginated until reaching the end or max items limit
• All timestamps are in ISO 8601 format
• PDF URLs may be null if the article doesn't have a publicly available PDF
• Open access status helps identify freely available articles
• Author information is extracted as an array for easy processing

---

Start scraping academic articles from ScienceDirect today!