Research Paper RAG: arXiv, PubMed and Open Access

What is Research Paper RAG for arXiv, PubMed and open access?

Research Paper RAG turns an open-access research paper into text an LLM pipeline can cite. Give it a DOI, a PMID, a PMCID, an arXiv id, an OpenAlex id, or a short search, and you get back:

• one row per paper (recordType: "document") with the title, authors, year, DOI, licence, and — when the licence allows it — the full text as Markdown;
• one row per chunk (recordType: "chunk") with the chunk text, its token count, the heading it came from, and the same citation (DOI, authors, licence) so a vector store can quote the paper without joining the rows.

It reads three official sources and nothing else. Europe PMC supplies the open-access full text of PubMed Central. The arXiv API supplies metadata, and arXiv HTML supplies the body only when that paper's own licence allows redistribution. OpenAlex supplies the licence, the identifiers and the open-access file link; its metadata is CC0. There is no browser, no login and no API key for the default path.

The prefilled run reads two papers: the FAIR Guiding Principles (PMC4792175, CC BY 4.0, from Europe PMC) and an arXiv review (arXiv:2405.13832, CC BY 4.0, from arXiv HTML).

Quick answer: when should you (or an AI agent) pick this Actor?

Pick this Actor when you are building retrieval over the scholarly literature and you need chunks plus a real citation: a research-paper Q&A agent, a literature review index, a nightly job that turns a DOI list into vectors, or a weekly PubMed / arXiv monitor that should only keep text you are allowed to store. Filter on license and on commercialUseAllowed before you embed.

Pick something else when the paper is paywalled, the PDF is a scan, or you already have the PDF file. This Actor does not log in, does not pass a captcha, and does not OCR. A scanned PDF comes back with no_text_layer and is not charged. For a PDF you already have a link to, use PDF to Markdown & RAG Chunks. For a scan, use Image to Text & Scanned PDF OCR. This Actor does not scrape Google Scholar.

Defaults that work: leave the two prefilled ids and click Start. You get both papers, their Markdown, and chunks of about 512 tokens. For a vector store, set What to return to Chunks only.

At a glance

Does	Reads open-access research papers and returns Markdown, token counts and citable RAG chunks
Input	DOIs, PMIDs, PMCIDs, arXiv ids, OpenAlex ids, a search query, a dataset of those ids, or a weekly keyword watch
Sources	Europe PMC full text (PubMed OA), arXiv HTML, OpenAlex metadata and open-access files, plus an optional Unpaywall PDF
Output	document rows (citation, licence, Markdown, status) and chunk rows (text, tokens, headings, DOI, licence)
Needs	No login and no API key. An OpenAlex key is optional, and only for OpenAlex search or its cached PDF
Pricing	Pay per event: $0.012 per research paper on the Free plan, plus a small start fee. Skipped and failed papers are free. Chunks are free
Does not	Scrape Google Scholar, log in, bypass a block, OCR a scan, or return a body whose licence does not allow redistribution

Key terms used in this README

• Research paper: a journal article or an arXiv preprint identified by a DOI, PMID, PMCID, arXiv id or OpenAlex id. One processed paper is one charge, however many chunks it becomes.
• Open access: the paper carries a licence that allows the text to be redistributed. Creative Commons (CC BY, CC BY-SA, CC BY-NC, CC BY-ND and the combined NC/ND forms), CC0 and public domain qualify. The default arXiv non-exclusive licence does not: those preprints stay on arXiv, and this Actor returns metadata only.
• Document row: recordType: "document". One per paper, including skips and failures, so a miss has a reason instead of a hole.
• Chunk row: recordType: "chunk". A slice of about chunkSize tokens. The fields match the PDF RAG Actor in this account (text, tokenCount, pageStart, pageEnd, headings, isPartial, chunkIndex, documentTitle, url) and add doi, author, license and citation.
• Token count: counted with cl100k_base, the tokenizer used by OpenAI embedding and chat models, not estimated from characters.
• Licence flags: commercialUseAllowed is false for NC. derivativesAllowed is false for ND. ND text is still returned, because the licence is an open licence; read the flag before you adapt it.

What can you use it for?

RAG over the literature

mode: "chunks" is an embedding input. Store doi, license, headings and citation next to the vector. An answer can say "Wilkinson et al., 2016, CC BY 4.0" instead of "a paper we found".

A DOI list from somewhere else

Point Read identifiers from a dataset at a dataset that already has a doi or pmcid column. The Actor opens it read-only, dedupes the ids and reads the ones that are open.

PubMed and Europe PMC full text

Europe PMC's API is the open-access full text of PubMed Central, as JATS XML. The Actor turns sections into Markdown headings and drops the reference list unless you ask for it. Author emails in the XML are removed.

arXiv, when the author chose a reusable licence

Most arXiv preprints use the arXiv non-exclusive licence, which lets arXiv distribute the file and does not let a third party republish the text. Those come back as license_not_redistributable, not charged, with the API abstract (arXiv publishes abstracts as CC0 metadata). Papers submitted as CC BY, CC BY-SA, CC BY-NC, CC0 or public domain are read from the official HTML (arxiv.org/html/…), which arXiv's robots.txt allows.

OpenAlex as the catalogue

OpenAlex metadata is CC0. A single-record lookup (by DOI, PMID, arXiv id or work id) needs no key. Use it to learn the licence and the open-access PDF link. Keyword search on OpenAlex, and OpenAlex's cached PDF, need your API key because OpenAlex meters those calls. This Actor never uses a shared key.

New papers this week

Turn on New papers this week, put a keyword in Search query, clear the Papers list (the two examples would otherwise be read on every run), and schedule the Actor weekly. Europe PMC (PubMed open access, FIRST_PDATE) and the arXiv API (submittedDate) return papers first published in the last 7 UTC days. OpenAlex is not searched in this mode, because its keyword search needs your key. The run looks at up to 100 hits per source and keeps at most maxPapers that this query has not already returned.

Papers already returned for that query are stored in the named key-value store research-paper-rag-watch and are not charged again. A week with nothing new succeeds, so a schedule does not look broken. A download that fails still fails the run. arXiv HTML is opened during a watch so the licence box can be read; the body is kept only when that box allows redistribution. A normal search still does not open arXiv HTML for a hit whose licence is unknown.

The same Research paper event applies. There is no separate watch charge. Skipped and failed papers stay free.

A DOI whose copy is only on Unpaywall

If Europe PMC and arXiv have no redistributable full text, and you set Email for Unpaywall to your own address, the Actor asks Unpaywall's free API for a PDF. The PDF is downloaded only when that location's licence allows redistribution. A missing licence, a publisher-specific licence, and an HTML landing page are not used. The email is the parameter Unpaywall requires. It is not written to the dataset, the summary or the log. Placeholder domains such as example.com are rejected.

Tables and figures

A table caption, the table body, and a figure caption are each their own chunk. They are not packed into the paragraph before or after them, which is where a number in a table would otherwise disappear into the methods section. Chunks are still free.

How does it work?

1. Each input is parsed into a DOI, PMID, PMCID, arXiv id or OpenAlex id. Links with a username and password are rejected. Duplicates collapse to one paper.
2. Metadata is read from the sources you enabled. Europe PMC and the arXiv API need no key. OpenAlex single-record lookups need no key.
3. The licence decides the full text, before a file is downloaded when the licence is already known:
   • Europe PMC XML when the paper is in the open-access subset;
   • arXiv HTML when the licence allows redistribution, or when you named that arXiv id (the page's licence box is then checked, and the body is dropped if it does not qualify);
   • an OpenAlex cached PDF only if you supplied a key;
   • otherwise the open-access PDF link, and only with a redistributable licence;
   • if those are empty and you gave an email, a Unpaywall PDF whose own licence allows redistribution.
4. A 403 or 429 stops that host for the rest of the run. The request is not retried and no proxy is used to get around it. (OpenAlex documents that a per-second 429 should be waited out once; that single wait is the only retry.)
5. Sections become Markdown. Paragraphs are packed into chunks of chunkSize tokens with chunkOverlap tokens repeated. A table caption, the table itself and a figure caption each stay in their own chunk. A paragraph longer than the budget is split on sentence ends.
6. The paper row is charged once when the body is returned. Chunks are free. A skip or a failure is a row and costs nothing. A weekly watch charges that same event, and only for a paper it has not returned before.

Requests are spaced to the published limits: 10 seconds on www.ebi.ac.uk (its robots.txt crawl delay), 3 seconds on the arXiv API, 15 seconds on arxiv.org (robots.txt crawl delay for /html and /pdf), under one request a second on OpenAlex, and one request a second on Unpaywall.

How to extract a research paper (step by step)

1. Open the Actor. The input already contains PMC4792175 and arXiv:2405.13832.
2. Leave What to return on Paper and chunks for a first look, or choose Chunks only for embeddings.
3. Leave Chunk size at 512. Use 256 for narrow facts.
4. Leave Which licences to read on Any open licence. Choose Commercial use only to skip NC licences.
5. Click Start. Open Papers for the citation and the status, or Chunks for the embedding rows.
6. Download JSON, JSONL, CSV or Excel. recordType separates the two kinds of row.

Input

Field	Type	Default	What it does
papers	list of text	two example papers	DOIs, PMIDs, PMCIDs, arXiv ids, OpenAlex ids, or links to them. Up to 25
query	text	empty	Search. Europe PMC is limited to open-access records that have full text. A normal arXiv search fetches HTML only when a redistributable licence is already known. Required when the weekly watch is on
sources	list	all three	europepmc, openalex, arxiv
maxPapers	number	5	Cap on papers per run, after deduplication
datasetId	text	empty	Read ids from another dataset
watchNewPapers	yes/no	no	Keyword watch: Europe PMC and arXiv, papers from the last 7 days. Schedule it weekly. Already-returned papers are free
mode	choice	both	both, chunks or document
chunkSize	number	512	Token budget, 64–4096
chunkOverlap	number	64	Repeated tokens, at most half of chunkSize
includeMarkdown	yes/no	yes	markdown on the paper row
includeReferences	yes/no	no	Keep the bibliography. Off, because it crowds out the paper
maxChunksPerPaper	number	400	Safety cap. Chunks are not billed
licenseFilter	choice	any-open	commercial skips NC licences
openAlexApiKey	secret	empty	Your OpenAlex key. Optional. Not used by the weekly watch
contactEmail	secret	empty	Your email for the Unpaywall API. Optional. Not stored

Output

Paper row (shape; the live example is in the screenshot below):

{
  "recordType": "document",
  "url": "https://doi.org/10.1038/sdata.2016.18",
  "status": "processed",
  "title": "The FAIR Guiding Principles for scientific data management and stewardship",
  "author": "Mark D. Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Bonino da Silva Santos, Philip E. Bourne, Jildau Bouwman, Anthony J. Brookes",
  "year": 2016,
  "doi": "10.1038/sdata.2016.18",
  "pmcid": "PMC4792175",
  "license": "CC BY 4.0",
  "licenseUrl": "https://creativecommons.org/licenses/by/4.0/",
  "commercialUseAllowed": true,
  "derivativesAllowed": true,
  "source": "europepmc",
  "fullTextFormat": "jats",
  "tokenCount": 6006,
  "chunkCount": 15,
  "citation": "Mark D. Wilkinson et al. (2016). The FAIR Guiding Principles for scientific data management and stewardship. https://doi.org/10.1038/sdata.2016.18 License: CC BY 4.0."
}

Chunk row:

{
  "recordType": "chunk",
  "url": "https://doi.org/10.1038/sdata.2016.18",
  "documentTitle": "The FAIR Guiding Principles for scientific data management and stewardship",
  "chunkIndex": 0,
  "text": "## Abstract\n\nThere is an urgent need to improve the infrastructure supporting the reuse of scholarly data. A diverse set of stakeholders—rep…",
  "tokenCount": 431,
  "pageStart": null,
  "pageEnd": null,
  "headings": ["Abstract", "Comment", "Supporting discovery through good data management"],
  "isPartial": false,
  "doi": "10.1038/sdata.2016.18",
  "license": "CC BY 4.0",
  "citation": "Mark D. Wilkinson et al. (2016). The FAIR Guiding Principles for scientific data management and stewardship. https://doi.org/10.1038/sdata.2016.18 License: CC BY 4.0."
}

The chunk text above is shortened so the page stays readable. tokenCount 431 is the full first chunk from the measured run, not the length of that excerpt.

pageStart and pageEnd are null for XML and HTML, which have no pages. They are filled for a PDF. headings is the citation inside the paper either way. The same chunk fields (text, tokenCount, charCount, pageStart, pageEnd, headings, isPartial, chunkIndex, documentTitle, url, recordType) are what PDF to Markdown & RAG Chunks writes, so one embedding job can read both datasets.

status: "skipped" with failureReason: "license_not_redistributable" or "not_open_access" means the body was not stored and you were not charged. blocked means the host returned 403 or 429 and the Actor stopped calling it.

How much does a research paper cost?

Pay per event. You pay for a paper whose full text was returned, plus a small start fee. Chunk size does not change the price. A skipped paper, a failed download, a scan and a non-redistributable preprint are free.

Event	Free	Bronze	Silver	Gold and above
Actor start (once per GB of memory, minimum one)	$0.0005	$0.0005	$0.0005	$0.0005
Research paper (primary)	$0.012	$0.0108	$0.0096	$0.0084

Ten papers on the Free plan cost about $0.12 plus the start fee (under a cent at 512 MB). The Apify free plan's monthly credit covers a few hundred papers. Raising the chunk count does not raise the bill. maxPapers is the cost cap. A weekly watch uses the same event: you pay for a new open-access paper, not for the search and not for a paper the watch has already returned.

ScholarAPI publishes a starter pack of 2,000 credits for $29.90, which is $0.015 per credit. Their full-text call (GET /text) is listed at 5 credits, about $0.075 a paper on that pack. A promotional 3-credit rate on that endpoint runs through 31 December 2026. This Actor's Free-plan list price for one research paper whose full text is returned is $0.012, chunks included. Those ScholarAPI figures were read from their credits page on 4 November 2026 and can change. This Actor is not ScholarAPI and does not call it.

Platform cost was measured on Apify before this price was set. The two prefilled papers, on build 1.0.1 at 512 MB (run RAkJGcWTgeV25arsf), finished in 5.5 seconds. Finalized platform usage was $0.00051 for both papers, about $0.00026 each: compute units $0.00015 and 50 dataset writes $0.00025. The Free-plan price of $0.012 is about 47 times that cost. If your OpenAlex key downloads a cached PDF, OpenAlex also bills your key about $0.01 for that file; that is OpenAlex's price, not an extra Apify event.

Tips for best results

• Prefer a PMCID or a DOI. Europe PMC XML keeps the section headings, which makes better chunks than a PDF.
• Leave references off. Turn them on only when the bibliography is the thing you want to search.
• For a commercial product, set Which licences to read to Commercial use only, and still keep license on the chunk. CC BY wants attribution; the citation field is there for that.
• arXiv HTML exists for many papers from 2024 on, not for every older preprint. A paper with no HTML and no redistributable licence is skipped, not guessed from the PDF. A weekly watch opens that HTML, so five arXiv papers take a little over a minute because of the 15-second crawl delay.
• PDFs are capped at 15 MB and 40 pages. A longer open PDF is marked truncated and still charged once, because the charge is per paper.
• If a host returns 403 or 429, that host is done for the run. Split the list and run it later rather than expecting a retry.

Integrations and API

Run it from the Apify API, on a schedule, or from an AI agent through MCP. Send the dataset to a vector database, Google Sheets, Slack, Make, Zapier or n8n. The chunk view is the one to embed. The paper view is the one to audit licences.

curl "https://api.apify.com/v2/acts/mmdincer~research-paper-rag/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"papers":["PMC4792175"],"mode":"chunks","chunkSize":512}'

Is it legal to read these research papers?

This Actor uses official machine interfaces and returns full text only under the paper's own licence.

Europe PMC. Full text is the REST API on www.ebi.ac.uk (/europepmc/webservices/rest/{PMCID}/fullTextXML). That path is not disallowed by EBI's robots.txt. The crawl delay of 10 seconds for User-agent: * is honoured. Articles in the open-access subset carry a licence, often with a text-mining reference, and that licence is copied onto every row. The Europe PMC website sits behind a Cloudflare check and its robots file sends a content signal of ai-train=no; this Actor does not scrape the website. NCBI describes the PMC open-access article set as released for reuse under the article licences.

arXiv. The API terms (info.arxiv.org) say metadata, including the abstract, is CC0, and that storing or serving the e-print itself is allowed only when the licence the submitter chose permits redistribution. They also say most e-prints use the arXiv non-exclusive licence, which does not. This Actor follows that line: non-exclusive and unknown licences do not get a body. HTML is fetched from arxiv.org/html/, which robots.txt allows, with the 15-second crawl delay. The API is called at most once every three seconds. arXiv is not the copyright holder of the papers.

OpenAlex. The catalogue is CC0 ("free data, paid service"). Single-record lookups are free without a key. Search and cached-PDF downloads are metered by OpenAlex, so they run only with a key you provide, and they spend your allowance. This Actor does not ship a key of its own. The weekly watch does not call OpenAlex search.

Unpaywall. The lookup is their free API (api.unpaywall.org/v2/{doi}). It requires your email in the query and asks callers not to use a shared or placeholder address. This Actor sends the address you type, at most one request a second, and does not store it. Only url_for_pdf is downloaded, and only when that location's licence allows redistribution. Publisher HTML is not scraped.

Your obligations. You decide what you do with the text afterwards. NC means no commercial use. ND may not allow an adapted copy; the row says so. CC BY requires attribution, which the citation field provides. Do not use this for paywalled full text. If a rightsholder asks, the Actor will be taken down; use the Issues tab on this page.

Not affiliated with arXiv, Europe PMC, PubMed, NCBI, OpenAlex, Unpaywall, Crossref, ScholarAPI, or any journal or publisher. Names are used only to say where the text came from.

FAQ

Does it need a key? No. Add an OpenAlex key only if you want OpenAlex keyword search or OpenAlex's cached PDF. The weekly watch and Unpaywall do not use that key. Unpaywall needs your email, not a key.

How do I watch for new papers? Clear Papers, set New papers this week, put the keyword in Search query, leave Maximum papers at the number you are willing to pay for, and schedule the Actor weekly. Europe PMC and arXiv only. A quiet week is a successful run with no charge beyond the start fee.

Does it scrape Google Scholar? No. Scholar is not a source.

Why was my arXiv paper skipped? The HTML licence box was not a Creative Commons or CC0 licence. That is the usual arXiv licence, and the text stays on arXiv. You are not charged.

Why are page numbers empty? XML and HTML are not paginated. Use headings. PDF rows have pageStart and pageEnd.

Why did a search return fewer papers than I asked for? maxPapers caps the run, duplicates are removed, and papers without a redistributable full text are skipped rather than padded.

Is a failed paper billed? No. Only status: "processed" is a Research paper event.

How do I report a problem? Open an issue on this Actor with the run id and the paper id.

Other Actors

Actor	What it does
PDF to Markdown & RAG Chunks	The same chunk shape for a PDF file you already have a link to
Image to Text & Scanned PDF OCR	OCR when the PDF is a scan and has no text layer
GitHub Repo RAG	A repository as token-counted chunks, with the repo licence