Academic Research Paper Scraper: ArXiv, PubMed & Crossref DOIs

Scrape and deduplicate academic research papers, preprints, abstracts, citation counts, DOIs, and Open Access PDF links from ArXiv, PubMed/EuropePMC, and Crossref.

What it does

Queries ArXiv, EuropePMC/PubMed, and Crossref in parallel, cross-deduplicates by DOI/title, and outputs clean abstracts, authors, citation counts, and PDF URLs. Export clean, deduplicated data as JSON, CSV, Excel, XML, or RSS, or connect directly via the Apify REST API, Python/JavaScript SDKs, CLI, MCP Server, n8n, Make, or Zapier — with no login, cookies, or browser overhead.

Use cases

• AI Agent Literature Review & Scientific RAG Pipelines: Feed deduplicated title, abstract, authors, citationCount, doi, and direct pdfUrl into LangChain, LlamaIndex, or vector databases.
• Biotech, Pharma & Clinical R&D Monitoring: Combine PubMed/EuropePMC (pmid, pmcid) with Crossref journal citations to track new CRISPR, oncology, or clinical trial publications.
• Machine Learning Preprint & Citation Tracking: Monitor ArXiv CS/AI preprints alongside peer-reviewed journal DOIs and citation velocity.
• Scheduled Literature Alerts: Run daily or weekly with onlyNew: true to alert research teams whenever new papers match target keywords.

Why use this scraper?

Feature	This Actor	Generic Browser / HTML Scrapers
Architecture	Direct structured API / JSON / feed requests (curl_cffi)	Heavy headless browser automation
Speed & reliability	2–25 seconds for 50–200 items	Minutes of high compute burn
Pricing model	Pay per result ($0.00 actor-start fee)	Monthly rental or high CU overhead
Error handling	Failed inputs logged in ERRORS key-value record and never charged	Bills compute time even on failures

Input parameters

• Research search queries (queries, array): Keywords, topics, author names, or DOIs to search across ArXiv, EuropePMC/PubMed, and Crossref.
• Academic databases to query (sources, array): Any combination of 'arxiv', 'europepmc' (PubMed / PMC biomedical literature), and 'crossref' (DOI journal registry).
• Minimum publication year (minYear, integer): Filter out papers published before this year.
• Sort order (sortBy, string): Sort results by relevance, publication date, or citation count.
• Open-access / free PDF only (openAccessOnly, boolean): Emit only papers that are marked Open Access or include a direct public PDF URL.
• Only new papers (onlyNew, boolean): Remember seen paper DOIs/IDs across runs in Key-Value Store and emit only newly published papers.
• Max papers per query (maxPapersPerQuery, integer): Maximum number of deduplicated papers to return per search query.
• Delay between requests (delaySeconds, number): Polite pause between requests (default 0.4).

Example input

{
  "queries": [
    "large language models",
    "crispr gene editing"
  ],
  "sources": [
    "europepmc",
    "arxiv",
    "crossref"
  ],
  "minYear": 2020,
  "maxPapersPerQuery": 35
}

Output & example output

One dataset row per result. Default table view columns: source, title, authors, venue, publicationYear, citationCount, isOpenAccess, pdfUrl, url. Full structured JSON is stored in the run's default dataset.

Example output item

{
  "paperId": "pmid:42808869",
  "source": "europepmc",
  "query": "large language models",
  "title": "Large Language Models as Diagnostic Tools in Nephrology: Helpful, With Limitations.",
  "abstract": "Background Large language models (LLMs) are increasingly being used to support diagnostic decision-making. It is unclear, however, whether their performance on knowledge and exa...",
  "authors": [
    "Hasan C",
    "Boeckel GR",
    "Siam S"
  ],
  "authorCount": 6,
  "publicationDate": "2026-12-11",
  "publicationYear": 2026,
  "venue": "Deutsches Arzteblatt international",
  "doi": "10.3238/arztebl.m2026.0164",
  "arxivId": null,
  "pmid": "42808869",
  "pmcid": null,
  "citationCount": 0,
  "isOpenAccess": false,
  "pdfUrl": null,
  "url": "https://pubmed.ncbi.nlm.nih.gov/42808869/",
  "subjects": [],
  "scrapedAt": "2026-10-25T11:51:54+00:00"
}

Pricing (Pay-Per-Result)

This Actor uses transparent Pay-Per-Event pricing with a $0.00 actor-start fee:

Event	What is charged	Price
paper-result	Research paper scraped (One deduplicated academic paper or preprint with abstract, citations & PDF link)	$0.002 / result ($2.00 / 1,000 results)
apify-actor-start	Actor start fee	$0.00

Concrete pricing math

• paper-result (Research paper scraped): 1,000 results = $2.00 (50 results = $0.10, 10,000 results = $20.00). Apify's $5.00/month Free Tier covers up to 2,500 results/month at $0.00 out-of-pocket.
• Empty runs (0 results) & skipped duplicates (onlyNew / onlyChanged): $0.00.

Data and compliance

Public open-science APIs (ArXiv export API, EuropePMC REST API, Crossref polite pool) only. Zero paywall circumvention or personal data. Requests are rate-limited and routed through Apify Proxy. Per-input errors are recorded in the run's ERRORS key-value record and are never charged.

FAQ

Do I need an account, login, or API key?
No. This Actor queries public endpoints directly and requires zero credentials.

What happens if I start a run with empty {} input via API or CLI?
Every required input defines both prefill and default values in .actor/input_schema.json, so an empty {} run automatically returns 50+ live records out of the box.