Academic Papers Scraper – arXiv, PubMed, OpenAlex & Crossref

Search academic papers in arXiv, PubMed, OpenAlex, Crossref and Semantic Scholar with one query - a Google Scholar alternative with no captchas. Get titles, authors, abstracts, DOIs, citation counts, venues and open-access PDF links. Filter by year, sort by citations, look up DOI lists.

Price: $1 per 1,000 papers. Failed sources or unknown DOIs are never charged.

What data do you get?

One row per paper, with the same fields no matter which database it came from:

• Title, authors, year, publication date, type (article, preprint, review, conference paper, book chapter, clinical trial…)
• Venue (journal / conference / repository) and publisher
• Abstract (rebuilt from OpenAlex's inverted index, cleaned from Crossref JATS XML, structured PubMed abstracts)
• Citation count (OpenAlex, Crossref, Semantic Scholar)
• Open access flag and direct PDF link (pdfUrl) when a free full text exists
• DOI, landing-page URL, arXiv ID, PubMed ID (PMID) and PubMed Central ID (PMCID)
• Topics – OpenAlex topics & keywords, MeSH terms, arXiv categories or fields of study
• sources – every database where the paper was found (duplicates are merged into one row, filling missing fields from the other sources)

Why use this instead of a Google Scholar scraper?

• No captchas, no blocked runs – Google Scholar has no API and blocks scrapers. This Actor uses the official, free APIs of five large open scholarly databases, so a default run takes about 5 seconds.
• Richer data – abstracts, open-access PDF links, PMIDs, topics and publication types that Scholar result pages don't show.
• Low, predictable cost – $1 per 1,000 papers, no start fee.
• One query, five databases, no duplicates – the same paper found in OpenAlex, Crossref and Semantic Scholar becomes one row.

Source	Best for	Size
OpenAlex	All fields, citations, open access, topics	250M+ works
arXiv	Preprints in CS/AI, physics, maths, statistics	2.5M+ preprints
Crossref	Every DOI from every publisher	170M+ records
PubMed	Medicine and life sciences	38M+ citations
Semantic Scholar	AI-indexed literature, citation counts	200M+ papers

Use cases

• Literature reviews & systematic reviews – collect every paper on a topic with abstracts, then screen them in a spreadsheet.
• AI agents and RAG – give your LLM agent a reliable "search papers" tool with abstracts and PDF links.
• Citation & bibliometric analysis – find the most cited papers on a topic in a year range.
• Research monitoring – schedule a daily run sorted by date to catch new preprints and publications.
• DOI enrichment – turn a list of DOIs into full metadata (authors, abstract, citations, open-access PDF).
• Field mapping – see which authors, venues and publishers are active on a topic.

How to use it

1. Enter a Search query (e.g. crispr gene editing).
2. Choose the Sources (all five by default), Max papers per source and Sort by (relevance, date or citations).
3. Optionally set Published from/until year and Open access only.
4. Click Start and export to JSON, CSV, Excel, or use the API.

DOI lookup mode: paste DOIs into DOIs to look up instead of a query. Each DOI is looked up in all selected sources at once and returned as one merged record.

Input

Field	Type	Description
query	string	Keywords to search. Default large language models. arXiv syntax (ti:, au:, cat:) is passed through to arXiv.
sources	array	Any of openalex, arxiv, crossref, pubmed, semanticScholar. Default: all.
maxItems	integer	Max papers per source (1–1000). Default 10.
sortBy	string	relevance (default), date (newest first) or citations (most cited first; arXiv and PubMed fall back to relevance).
yearFrom / yearTo	integer	Publication year range.
openAccessOnly	boolean	Only free-to-read papers. Default false.
dois	array	DOI lookup mode: 10.1038/nature14539 or https://doi.org/....
dedupe	boolean	Merge the same paper from several sources. Default true.
contactEmail	string	Optional. Sent as mailto to OpenAlex/Crossref/PubMed for their faster "polite pool". Omitted if empty.
semanticScholarApiKey	string (secret)	Optional free key for faster, relevance-ranked Semantic Scholar results.

Example input:

{
    "query": "crispr gene editing",
    "sources": ["openalex", "pubmed", "semanticScholar"],
    "maxItems": 50,
    "sortBy": "citations",
    "yearFrom": 2019,
    "yearTo": 2024,
    "openAccessOnly": true
}

DOI lookup:

{
    "dois": ["10.1038/nature14539", "https://doi.org/10.1056/NEJMoa2032183"]
}

Output

Real sample (DOI lookup of 10.1038/nature14539, shortened):

{
    "itemType": "paper",
    "source": "openalex",
    "sources": ["openalex", "crossref", "pubmed"],
    "id": "W2919115771",
    "doi": "10.1038/nature14539",
    "title": "Deep learning",
    "authors": ["Yann LeCun", "Yoshua Bengio", "Geoffrey E. Hinton"],
    "year": 2015,
    "publicationDate": "2015-05-26",
    "venue": "Nature",
    "publisher": "Nature Portfolio",
    "abstract": "Deep learning allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction. ...",
    "citationCount": 85418,
    "openAccess": true,
    "pdfUrl": null,
    "url": "https://doi.org/10.1038/nature14539",
    "topics": ["Neural Networks and Applications", "deep learning", "backpropagation", "convolutional neural networks"],
    "type": "article",
    "arxivId": null,
    "pmid": "26017442",
    "pmcid": null,
    "language": "en",
    "query": "10.1038/nature14539",
    "scrapedAt": "2026-10-07T19:08:34.711Z"
}

An arXiv preprint looks the same, with "source": "arxiv", "type": "preprint", "openAccess": true, "pdfUrl": "https://arxiv.org/pdf/2402.14679v2" and "citationCount": null (arXiv has no citation data – enable OpenAlex or Semantic Scholar to fill it in when the paper is found there too).

Free error / status rows (itemType: "error" or "status") tell you when a source was rate-limited (RATE_LIMITED), failed (SOURCE_FAILED), a DOI was invalid (INVALID_DOI) or not found (NOT_FOUND), or a search had no results (NO_RESULTS). They are never charged.

The dataset has ready-made views: Overview, Abstracts & topics, Identifiers and Errors & status.

Pricing

Pay-per-event – you only pay for papers saved. Platform usage is included.

Event	Price	When it is charged
paper	$0.001 per paper ($1 / 1,000)	Each paper saved to the dataset (after duplicates are merged)
Errors / status rows	Free	Failed sources, invalid or unknown DOIs, empty searches

Job	Cost
Default run (10 papers × 5 sources, minus duplicates)	≈ $0.05
100 papers from each of 5 sources	≤ $0.50
Look up 1,000 DOIs	≤ $1.00

Set Max cost per run in the run options – the Actor stops exactly at your budget.

Use with AI agents (MCP)

This Actor works as a tool for AI agents through the Apify MCP server (Claude, ChatGPT, Cursor, VS Code and other MCP clients). Add it with https://mcp.apify.com?tools=openharvest/academic-papers-scraper, or let the agent find it with search-actors ("academic papers", "Google Scholar", "arXiv", "PubMed").

Example call-actor input an agent can send:

{
    "query": "retrieval augmented generation evaluation",
    "sources": ["openalex", "arxiv", "semanticScholar"],
    "maxItems": 10,
    "sortBy": "citations",
    "yearFrom": 2023
}

The agent gets flat JSON with title, authors, year, abstract, citationCount, doi, pdfUrl and url – ready to summarise or cite. Default runs finish in seconds.

FAQ

Is this a Google Scholar scraper? No – it does not touch Google Scholar (which has no API and blocks bots). It searches the open databases most of Scholar's content comes from, through their official APIs, so it never hits captchas.

Why are some citation counts null? arXiv and PubMed don't publish citation counts. With deduplication on, a paper also found in OpenAlex or Semantic Scholar gets their citation count.

Why do some open-access papers have no pdfUrl? Some free papers are only available as a web page (e.g. PubMed Central). Use url, or the pmcid (https://pmc.ncbi.nlm.nih.gov/articles/<pmcid>/).

Semantic Scholar shows a RATE_LIMITED row – what now? Semantic Scholar's public API is shared by everyone and is often busy. The Actor retries, then falls back to its bulk search (most cited first). If it is still busy you get a free error row and the other sources are unaffected. A free API key (input semanticScholarApiKey) solves it.

How does deduplication work? Papers with the same DOI, or the same normalized title, are merged. The first source's record is kept and empty fields are filled from the others (abstract, PDF link, highest citation count). sources lists every database where the paper appeared.

How many results can I get? Up to 1,000 per source per run (arXiv and PubMed allow more via several runs with year ranges).

Should I set the contact email? It's optional. OpenAlex, Crossref and NCBI ask API users to identify themselves with an email for their faster "polite pool". It's only sent to those APIs.

Is it legal?

The Actor uses the official public APIs of OpenAlex (CC0 data), arXiv, Crossref, NCBI PubMed and Semantic Scholar, within their documented rate limits, and collects bibliographic metadata only – no paywalled full texts and no personal contact data. Abstracts and texts remain the copyright of their authors/publishers; respect each source's terms (the arXiv API terms of use, the Semantic Scholar API license and the NCBI usage policies) when you reuse the data. If unsure, consult a lawyer.

Related actors

• Google Trends Scraper – Interest Over Time & Related Queries: see how interest in a research topic changes
• Hacker News Scraper – Stories, Comments & Who Is Hiring Jobs: discussion of new papers and tools
• Bluesky Scraper – Posts, Search, Profiles & Followers: posts by researchers and labs

Support & feedback

Missing a field or a source? Open an issue in the Issues tab – feature requests are welcome.