Academic Research Paper Scraper

Apify actor that scrapes academic papers from arXiv and PubMed with a unified output format.

Features

• Dual Source Support: Search both arXiv and PubMed simultaneously
• Unified Output: Consistent paper format regardless of source
• Smart Deduplication: Remove duplicates by DOI across sources
• Flexible Filtering: Filter by title, author, categories, and date range
• Rate Limit Compliance: Built-in throttling and automatic retry with exponential backoff
• PubMed API Key Support: Optional API key for faster PubMed access
• Open Access PDF Recovery: Optional Unpaywall lookup for papers missing a PDF link

Input Parameters

Parameter	Type	Required	Default	Description
searchQuery	string	Yes	-	Keywords, phrases, or terms to search
sources	array	No	["arxiv", "pubmed"]	Which databases to search
titleFilter	string	No	-	Filter papers by title keywords
authorFilter	string	No	-	Filter by author name
categories	array	No	-	arXiv categories (e.g., cs.AI, physics.quant-ph)
dateFrom	string	No	-	Start date (YYYY-MM-DD)
dateTo	string	No	-	End date (YYYY-MM-DD)
maxResults	integer	No	100	Max papers per source (1-10000)
sortBy	string	No	relevance	Sort order: relevance, date_desc, date_asc
pubmedApiKey	string	No	-	NCBI API key for faster rate limits
unpaywallEmail	string	No	-	Your email for Unpaywall API (free, no signup)
includeAbstract	boolean	No	true	Include full abstract text
deduplicateByDoi	boolean	No	true	Remove cross-source duplicates

Output Format

Each paper is pushed to the dataset as one record in a unified format. Not every field is present on every record — several are source-specific (see the table).

Example arXiv record:

{
  "id": "arxiv:2401.12345",
  "source": "arxiv",
  "arxivId": "2401.12345",
  "doi": "10.1234/example",
  "title": "Paper Title",
  "abstract": "Full abstract text...",
  "authors": [
    { "name": "John Doe" }
  ],
  "publishedDate": "2024-01-15",
  "updatedDate": "2024-01-20",
  "categories": ["cs.AI", "cs.LG"],
  "journal": "Nature",
  "abstractUrl": "http://arxiv.org/abs/2401.12345",
  "pdfUrl": "http://arxiv.org/pdf/2401.12345",
  "htmlUrl": "http://arxiv.org/abs/2401.12345",
  "comment": "10 pages, 4 figures"
}

Example PubMed record:

{
  "id": "pmid:39876543",
  "source": "pubmed",
  "pmid": "39876543",
  "pmcid": "PMC1234567",
  "doi": "10.5678/example",
  "title": "Paper Title",
  "abstract": "Full abstract text...",
  "authors": [
    { "name": "Jane Smith", "affiliation": "University Hospital" }
  ],
  "publishedDate": "2024-02-10",
  "categories": ["Neoplasms", "Gene Expression"],
  "journal": "Journal Name",
  "abstractUrl": "https://pubmed.ncbi.nlm.nih.gov/39876543/",
  "pdfUrl": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1234567/pdf/",
  "htmlUrl": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1234567/"
}

Output Fields

Field	Type	Populated for	Description
id	string	both	arxiv:<arxivId> or pmid:<pmid>
source	string	both	"arxiv" or "pubmed"
doi	string	when available	Digital Object Identifier
arxivId	string	arXiv only	arXiv identifier (e.g. 2401.12345)
pmid	string	PubMed only	PubMed ID
pmcid	string	PubMed only	PubMed Central ID (when deposited in PMC)
title	string	both	Paper title
abstract	string	both	Abstract text; empty string if includeAbstract=false or none exists
authors	array	both	{ name, affiliation? }; affiliation is PubMed-only
publishedDate	string	both	Publication date YYYY-MM-DD (may be empty if the source omits it)
updatedDate	string	arXiv only	Last update date YYYY-MM-DD
categories	array	both	arXiv taxonomy codes, or PubMed MeSH terms + author keywords
journal	string	when available	Journal name / arXiv journal reference
abstractUrl	string	both	Link to the abstract / landing page
pdfUrl	string	when available	Direct PDF link (see PDF Link Availability)
htmlUrl	string	when available	HTML version of the paper
comment	string	arXiv only	Author comments

If a run finds no matching papers, a single status record is pushed instead ({ "_type": "status", "status": "no_results", ... }) so the dataset is never empty. A run summary (counts, per-source errors, timestamp) is also written to the default key-value store under the summary key.

Example Usage

Search for AI papers

{
  "searchQuery": "transformer attention mechanism",
  "sources": ["arxiv", "pubmed"],
  "categories": ["cs.AI", "cs.LG", "cs.CL"],
  "maxResults": 50,
  "sortBy": "date_desc"
}

Search by author

{
  "searchQuery": "deep learning",
  "authorFilter": "Hinton",
  "dateFrom": "2020-01-01",
  "maxResults": 100
}

PubMed only with API key

{
  "searchQuery": "CRISPR gene editing",
  "sources": ["pubmed"],
  "pubmedApiKey": "your-ncbi-api-key",
  "maxResults": 500
}

arXiv Categories

Common categories you can filter by:

• Computer Science: cs.AI, cs.CL, cs.CV, cs.LG, cs.NE, cs.RO
• Physics: physics.quant-ph, physics.comp-ph
• Mathematics: math.OC, math.ST
• Statistics: stat.ML, stat.ME
• Quantitative Biology: q-bio.BM, q-bio.GN, q-bio.NC

Full taxonomy: https://arxiv.org/category_taxonomy

PDF Link Availability

Understanding why some papers lack PDF links:

Source	PDF Availability	Notes
arXiv	✅ Always available	arXiv is fully open access
PubMed with PMCID	✅ Available	Paper deposited in PubMed Central
PubMed without PMCID	⚠️ Often unavailable	Paywalled journal articles

Unpaywall Integration

When you provide unpaywallEmail, the actor queries Unpaywall to find open access versions of papers that lack PDF links. This can recover PDFs from:

• Institutional repositories
• Author preprint servers
• Publisher open access copies

Limitations:

• New papers (< 2-4 weeks old): Unpaywall may not have indexed them yet
• Paywalled papers with no OA version: No legal free PDF exists
• Papers without DOI: Cannot be looked up in Unpaywall

For recent papers without PDFs, the abstractUrl field always provides a link to the paper's landing page.

Rate Limits

The actor respects API rate limits:

• arXiv: 3-second delay between requests
• PubMed: 3 requests/second (or 10/second with API key)
• Unpaywall: 10 requests/second

Get a free PubMed API key at: https://www.ncbi.nlm.nih.gov/account/

Limitations

• Metadata + links only, no full text. The actor performs static HTTP calls to the arXiv and PubMed XML APIs (and optionally the Unpaywall JSON API). It parses metadata and link URLs; it does not render JavaScript, download PDF binaries, or extract full-text body content.
• maxResults is per source. It caps each of arXiv and PubMed independently (1–10000, default 100). With both sources enabled you may receive up to ~2× maxResults records before deduplication. Actual counts are lower when a source has fewer matches, and are bounded by what the upstream APIs return.
• Upstream rate limits (built-in throttling): arXiv ~1 request / 3 seconds; PubMed ~3 requests/second (10/second with an NCBI pubmedApiKey); Unpaywall ~10 requests/second. Large maxResults values therefore take longer to run — arXiv is paged at up to 2000 records/request, PubMed at up to 500/request.
• PubMed sort: both date_desc and date_asc map to PubMed's publication-date sort (newest first). True oldest-first ordering is only applied for arXiv.
• PDF availability is not guaranteed. arXiv records always include a PDF link. PubMed records only include a PDF link when a PMCID exists, or when Unpaywall finds an open-access copy. Paywalled articles with no OA version return no pdfUrl — use abstractUrl for the landing page.
• Unpaywall enrichment is opt-in. It runs only when you supply a valid unpaywallEmail, only for papers that are missing a pdfUrl and have a DOI. Very recent papers (< ~2–4 weeks) may not be indexed yet. No API key is required, but an email is mandatory per Unpaywall's terms.
• DOI-based deduplication only. deduplicateByDoi matches records across sources by DOI. Records without a DOI (common for arXiv preprints and some PubMed entries) cannot be cross-deduplicated.
• Author affiliations are PubMed-only. The arXiv API returns author names without affiliations.
• No results is a success, not a failure. If nothing matches (or both sources are transiently unavailable), the run still completes and pushes a single status record rather than failing; per-source errors are recorded in the run summary in the key-value store.

Data Sources

• arXiv API - Open access preprint server
• PubMed E-utilities - Biomedical literature database
• Unpaywall API - Open-access location lookup by DOI

License

MIT License