# Research Paper RAG: arXiv, PubMed and Open Access (`mmdincer/research-paper-rag`) Actor Turn an open-access research paper from arXiv, PubMed or Europe PMC into Markdown and token-counted RAG chunks. DOI, authors and license travel with every chunk. Official APIs only, no browser, no API key. Not affiliated with arXiv, PubMed, Europe PMC or OpenAlex. - **URL**: https://apify.com/mmdincer/research-paper-rag.md - **Developed by:** [Muhammet Mustafa Dincer](https://apify.com/mmdincer) (community) - **Categories:** AI, Developer tools, Automation - **Features:** Limited permissions - **Stats:** 1 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $8.40 / 1,000 research papers This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ![Open-access research paper to RAG chunks: one paper becomes a cited document and token-counted chunks](https://api.apify.com/v2/key-value-stores/vbQuJ5uhWVgUngr57/records/papers-banner.png?signature=OZSHuVycpJlIZTagQf6J) ### What is Research Paper RAG for arXiv, PubMed and open access? **Research Paper RAG** turns an open-access **research paper** into text an LLM pipeline can cite. Give it a DOI, a PMID, a PMCID, an arXiv id, an OpenAlex id, or a short search, and you get back: - **one row per paper** (`recordType: "document"`) with the title, authors, year, DOI, **licence**, and — when the licence allows it — the full text as Markdown; - **one row per chunk** (`recordType: "chunk"`) with the chunk text, its **token count**, the **heading** it came from, and the same citation (DOI, authors, licence) so a vector store can quote the paper without joining the rows. It reads three official sources and nothing else. **Europe PMC** supplies the open-access full text of PubMed Central. The **arXiv API** supplies metadata, and arXiv HTML supplies the body only when that paper's own licence allows redistribution. **OpenAlex** supplies the licence, the identifiers and the open-access file link; its metadata is CC0. There is no browser, no login and no API key for the default path. The prefilled run reads two papers: the FAIR Guiding Principles (`PMC4792175`, CC BY 4.0, from Europe PMC) and an arXiv review (`arXiv:2405.13832`, CC BY 4.0, from arXiv HTML). ### Quick answer: when should you (or an AI agent) pick this Actor? **Pick this Actor when** you are building retrieval over the scholarly literature and you need chunks plus a real citation: a research-paper Q\&A agent, a literature review index, a nightly job that turns a DOI list into vectors, or a weekly PubMed / arXiv monitor that should only keep text you are allowed to store. Filter on `license` and on `commercialUseAllowed` before you embed. **Pick something else when** the paper is paywalled, the PDF is a scan, or you already have the PDF file. This Actor does not log in, does not pass a captcha, and does not OCR. A scanned PDF comes back with `no_text_layer` and is not charged. For a PDF you already have a link to, use [PDF to Markdown & RAG Chunks](https://apify.com/mmdincer/pdf-rag-extractor). For a scan, use [Image to Text & Scanned PDF OCR](https://apify.com/mmdincer/image-ocr-rag). This Actor does not scrape Google Scholar. **Defaults that work:** leave the two prefilled ids and click **Start**. You get both papers, their Markdown, and chunks of about 512 tokens. For a vector store, set **What to return** to *Chunks only*. ### At a glance | | | |---|---| | **Does** | Reads open-access research papers and returns Markdown, token counts and citable RAG chunks | | **Input** | DOIs, PMIDs, PMCIDs, arXiv ids, OpenAlex ids, a search query, a dataset of those ids, or a weekly keyword watch | | **Sources** | Europe PMC full text (PubMed OA), arXiv HTML, OpenAlex metadata and open-access files, plus an optional Unpaywall PDF | | **Output** | `document` rows (citation, licence, Markdown, status) and `chunk` rows (text, tokens, headings, DOI, licence) | | **Needs** | No login and no API key. An OpenAlex key is optional, and only for OpenAlex search or its cached PDF | | **Pricing** | Pay per event: **$0.012 per research paper** on the Free plan, plus a small start fee. Skipped and failed papers are free. Chunks are free | | **Does not** | Scrape Google Scholar, log in, bypass a block, OCR a scan, or return a body whose licence does not allow redistribution | ### Key terms used in this README - **Research paper:** a journal article or an arXiv preprint identified by a DOI, PMID, PMCID, arXiv id or OpenAlex id. One processed paper is one charge, however many chunks it becomes. - **Open access:** the paper carries a licence that allows the text to be redistributed. Creative Commons (CC BY, CC BY-SA, CC BY-NC, CC BY-ND and the combined NC/ND forms), CC0 and public domain qualify. The default arXiv non-exclusive licence does not: those preprints stay on arXiv, and this Actor returns metadata only. - **Document row:** `recordType: "document"`. One per paper, including skips and failures, so a miss has a reason instead of a hole. - **Chunk row:** `recordType: "chunk"`. A slice of about `chunkSize` tokens. The fields match the PDF RAG Actor in this account (`text`, `tokenCount`, `pageStart`, `pageEnd`, `headings`, `isPartial`, `chunkIndex`, `documentTitle`, `url`) and add `doi`, `author`, `license` and `citation`. - **Token count:** counted with **cl100k_base**, the tokenizer used by OpenAI embedding and chat models, not estimated from characters. - **Licence flags:** `commercialUseAllowed` is false for NC. `derivativesAllowed` is false for ND. ND text is still returned, because the licence is an open licence; read the flag before you adapt it. ### What can you use it for? #### RAG over the literature `mode: "chunks"` is an embedding input. Store `doi`, `license`, `headings` and `citation` next to the vector. An answer can say "Wilkinson et al., 2016, CC BY 4.0" instead of "a paper we found". #### A DOI list from somewhere else Point **Read identifiers from a dataset** at a dataset that already has a `doi` or `pmcid` column. The Actor opens it read-only, dedupes the ids and reads the ones that are open. #### PubMed and Europe PMC full text Europe PMC's API is the open-access full text of PubMed Central, as JATS XML. The Actor turns sections into Markdown headings and drops the reference list unless you ask for it. Author emails in the XML are removed. #### arXiv, when the author chose a reusable licence Most arXiv preprints use the arXiv non-exclusive licence, which lets arXiv distribute the file and does not let a third party republish the text. Those come back as `license_not_redistributable`, not charged, with the API abstract (arXiv publishes abstracts as CC0 metadata). Papers submitted as CC BY, CC BY-SA, CC BY-NC, CC0 or public domain are read from the official HTML (`arxiv.org/html/…`), which arXiv's robots.txt allows. #### OpenAlex as the catalogue OpenAlex metadata is CC0. A single-record lookup (by DOI, PMID, arXiv id or work id) needs no key. Use it to learn the licence and the open-access PDF link. Keyword search on OpenAlex, and OpenAlex's cached PDF, need **your** API key because OpenAlex meters those calls. This Actor never uses a shared key. #### New papers this week Turn on **New papers this week**, put a keyword in **Search query**, clear the **Papers** list (the two examples would otherwise be read on every run), and schedule the Actor weekly. Europe PMC (PubMed open access, `FIRST_PDATE`) and the arXiv API (`submittedDate`) return papers first published in the last 7 UTC days. OpenAlex is not searched in this mode, because its keyword search needs your key. The run looks at up to 100 hits per source and keeps at most `maxPapers` that this query has not already returned. Papers already returned for that query are stored in the named key-value store `research-paper-rag-watch` and are not charged again. A week with nothing new succeeds, so a schedule does not look broken. A download that fails still fails the run. arXiv HTML is opened during a watch so the licence box can be read; the body is kept only when that box allows redistribution. A normal search still does not open arXiv HTML for a hit whose licence is unknown. The same **Research paper** event applies. There is no separate watch charge. Skipped and failed papers stay free. #### A DOI whose copy is only on Unpaywall If Europe PMC and arXiv have no redistributable full text, and you set **Email for Unpaywall** to your own address, the Actor asks Unpaywall's free API for a PDF. The PDF is downloaded only when that location's licence allows redistribution. A missing licence, a publisher-specific licence, and an HTML landing page are not used. The email is the parameter Unpaywall requires. It is not written to the dataset, the summary or the log. Placeholder domains such as `example.com` are rejected. #### Tables and figures A table caption, the table body, and a figure caption are each their own chunk. They are not packed into the paragraph before or after them, which is where a number in a table would otherwise disappear into the methods section. Chunks are still free. ### How does it work? 1. Each input is parsed into a DOI, PMID, PMCID, arXiv id or OpenAlex id. Links with a username and password are rejected. Duplicates collapse to one paper. 2. Metadata is read from the sources you enabled. Europe PMC and the arXiv API need no key. OpenAlex single-record lookups need no key. 3. The licence decides the full text, before a file is downloaded when the licence is already known: - Europe PMC XML when the paper is in the open-access subset; - arXiv HTML when the licence allows redistribution, or when you named that arXiv id (the page's licence box is then checked, and the body is dropped if it does not qualify); - an OpenAlex cached PDF only if you supplied a key; - otherwise the open-access PDF link, and only with a redistributable licence; - if those are empty and you gave an email, a Unpaywall PDF whose own licence allows redistribution. 4. A 403 or 429 stops that host for the rest of the run. The request is not retried and no proxy is used to get around it. (OpenAlex documents that a per-second 429 should be waited out once; that single wait is the only retry.) 5. Sections become Markdown. Paragraphs are packed into chunks of `chunkSize` tokens with `chunkOverlap` tokens repeated. A table caption, the table itself and a figure caption each stay in their own chunk. A paragraph longer than the budget is split on sentence ends. 6. The paper row is charged once when the body is returned. Chunks are free. A skip or a failure is a row and costs nothing. A weekly watch charges that same event, and only for a paper it has not returned before. Requests are spaced to the published limits: 10 seconds on `www.ebi.ac.uk` (its robots.txt crawl delay), 3 seconds on the arXiv API, 15 seconds on `arxiv.org` (robots.txt crawl delay for `/html` and `/pdf`), under one request a second on OpenAlex, and one request a second on Unpaywall. ### How to extract a research paper (step by step) 1. Open the Actor. The input already contains `PMC4792175` and `arXiv:2405.13832`. 2. Leave **What to return** on *Paper and chunks* for a first look, or choose *Chunks only* for embeddings. 3. Leave **Chunk size** at 512. Use 256 for narrow facts. 4. Leave **Which licences to read** on *Any open licence*. Choose *Commercial use only* to skip NC licences. 5. Click **Start**. Open **Papers** for the citation and the status, or **Chunks** for the embedding rows. 6. Download JSON, JSONL, CSV or Excel. `recordType` separates the two kinds of row. ### Input | Field | Type | Default | What it does | |---|---|---|---| | `papers` | list of text | two example papers | DOIs, PMIDs, PMCIDs, arXiv ids, OpenAlex ids, or links to them. Up to 25 | | `query` | text | empty | Search. Europe PMC is limited to open-access records that have full text. A normal arXiv search fetches HTML only when a redistributable licence is already known. Required when the weekly watch is on | | `sources` | list | all three | `europepmc`, `openalex`, `arxiv` | | `maxPapers` | number | 5 | Cap on papers per run, after deduplication | | `datasetId` | text | empty | Read ids from another dataset | | `watchNewPapers` | yes/no | no | Keyword watch: Europe PMC and arXiv, papers from the last 7 days. Schedule it weekly. Already-returned papers are free | | `mode` | choice | `both` | `both`, `chunks` or `document` | | `chunkSize` | number | 512 | Token budget, 64–4096 | | `chunkOverlap` | number | 64 | Repeated tokens, at most half of `chunkSize` | | `includeMarkdown` | yes/no | yes | `markdown` on the paper row | | `includeReferences` | yes/no | no | Keep the bibliography. Off, because it crowds out the paper | | `maxChunksPerPaper` | number | 400 | Safety cap. Chunks are not billed | | `licenseFilter` | choice | `any-open` | `commercial` skips NC licences | | `openAlexApiKey` | secret | empty | Your OpenAlex key. Optional. Not used by the weekly watch | | `contactEmail` | secret | empty | Your email for the Unpaywall API. Optional. Not stored | ### Output **Paper row** (shape; the live example is in the screenshot below): ```json { "recordType": "document", "url": "https://doi.org/10.1038/sdata.2016.18", "status": "processed", "title": "The FAIR Guiding Principles for scientific data management and stewardship", "author": "Mark D. Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Bonino da Silva Santos, Philip E. Bourne, Jildau Bouwman, Anthony J. Brookes", "year": 2016, "doi": "10.1038/sdata.2016.18", "pmcid": "PMC4792175", "license": "CC BY 4.0", "licenseUrl": "https://creativecommons.org/licenses/by/4.0/", "commercialUseAllowed": true, "derivativesAllowed": true, "source": "europepmc", "fullTextFormat": "jats", "tokenCount": 6006, "chunkCount": 15, "citation": "Mark D. Wilkinson et al. (2016). The FAIR Guiding Principles for scientific data management and stewardship. https://doi.org/10.1038/sdata.2016.18 License: CC BY 4.0." } ``` **Chunk row:** ```json { "recordType": "chunk", "url": "https://doi.org/10.1038/sdata.2016.18", "documentTitle": "The FAIR Guiding Principles for scientific data management and stewardship", "chunkIndex": 0, "text": "## Abstract\n\nThere is an urgent need to improve the infrastructure supporting the reuse of scholarly data. A diverse set of stakeholders—rep…", "tokenCount": 431, "pageStart": null, "pageEnd": null, "headings": ["Abstract", "Comment", "Supporting discovery through good data management"], "isPartial": false, "doi": "10.1038/sdata.2016.18", "license": "CC BY 4.0", "citation": "Mark D. Wilkinson et al. (2016). The FAIR Guiding Principles for scientific data management and stewardship. https://doi.org/10.1038/sdata.2016.18 License: CC BY 4.0." } ``` The chunk `text` above is shortened so the page stays readable. `tokenCount` 431 is the full first chunk from the measured run, not the length of that excerpt. `pageStart` and `pageEnd` are null for XML and HTML, which have no pages. They are filled for a PDF. `headings` is the citation inside the paper either way. The same chunk fields (`text`, `tokenCount`, `charCount`, `pageStart`, `pageEnd`, `headings`, `isPartial`, `chunkIndex`, `documentTitle`, `url`, `recordType`) are what [PDF to Markdown & RAG Chunks](https://apify.com/mmdincer/pdf-rag-extractor) writes, so one embedding job can read both datasets. `status: "skipped"` with `failureReason: "license_not_redistributable"` or `"not_open_access"` means the body was not stored and you were not charged. `blocked` means the host returned 403 or 429 and the Actor stopped calling it. ![Sample output: a paper row with its licence and the first chunks](https://api.apify.com/v2/key-value-stores/vbQuJ5uhWVgUngr57/records/papers-sample-output.png?signature=mxiXNxqV8tmpIf55POTw) ### How much does a research paper cost? Pay per event. You pay for a paper whose full text was returned, plus a small start fee. Chunk size does not change the price. A skipped paper, a failed download, a scan and a non-redistributable preprint are free. | Event | Free | Bronze | Silver | Gold and above | |---|---|---|---|---| | Actor start (once per GB of memory, minimum one) | $0.0005 | $0.0005 | $0.0005 | $0.0005 | | **Research paper** (primary) | **$0.012** | $0.0108 | $0.0096 | $0.0084 | Ten papers on the Free plan cost about **$0.12** plus the start fee (under a cent at 512 MB). The Apify free plan's monthly credit covers a few hundred papers. Raising the chunk count does not raise the bill. `maxPapers` is the cost cap. A weekly watch uses the same event: you pay for a new open-access paper, not for the search and not for a paper the watch has already returned. [ScholarAPI](https://scholarapi.net/credits) publishes a starter pack of 2,000 credits for $29.90, which is $0.015 per credit. Their full-text call (`GET /text`) is listed at 5 credits, about **$0.075** a paper on that pack. A promotional 3-credit rate on that endpoint runs through 31 December 2026. This Actor's Free-plan list price for one research paper whose full text is returned is **$0.012**, chunks included. Those ScholarAPI figures were read from their credits page on 4 November 2026 and can change. This Actor is not ScholarAPI and does not call it. Platform cost was measured on Apify before this price was set. The two prefilled papers, on build 1.0.1 at 512 MB (run `RAkJGcWTgeV25arsf`), finished in 5.5 seconds. Finalized platform usage was **$0.00051** for both papers, about **$0.00026** each: compute units $0.00015 and 50 dataset writes $0.00025. The Free-plan price of $0.012 is about 47 times that cost. If your OpenAlex key downloads a cached PDF, OpenAlex also bills **your** key about $0.01 for that file; that is OpenAlex's price, not an extra Apify event. ### Tips for best results - Prefer a PMCID or a DOI. Europe PMC XML keeps the section headings, which makes better chunks than a PDF. - Leave references off. Turn them on only when the bibliography is the thing you want to search. - For a commercial product, set **Which licences to read** to *Commercial use only*, and still keep `license` on the chunk. CC BY wants attribution; the `citation` field is there for that. - arXiv HTML exists for many papers from 2024 on, not for every older preprint. A paper with no HTML and no redistributable licence is skipped, not guessed from the PDF. A weekly watch opens that HTML, so five arXiv papers take a little over a minute because of the 15-second crawl delay. - PDFs are capped at 15 MB and 40 pages. A longer open PDF is marked `truncated` and still charged once, because the charge is per paper. - If a host returns 403 or 429, that host is done for the run. Split the list and run it later rather than expecting a retry. ### Integrations and API Run it from the Apify API, on a schedule, or from an AI agent through MCP. Send the dataset to a vector database, Google Sheets, Slack, Make, Zapier or n8n. The chunk view is the one to embed. The paper view is the one to audit licences. ```bash curl "https://api.apify.com/v2/acts/mmdincer~research-paper-rag/runs?token=YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{"papers":["PMC4792175"],"mode":"chunks","chunkSize":512}' ``` ### Is it legal to read these research papers? This Actor uses official machine interfaces and returns full text only under the paper's own licence. **Europe PMC.** Full text is the REST API on `www.ebi.ac.uk` (`/europepmc/webservices/rest/{PMCID}/fullTextXML`). That path is not disallowed by EBI's robots.txt. The crawl delay of 10 seconds for `User-agent: *` is honoured. Articles in the open-access subset carry a licence, often with a text-mining reference, and that licence is copied onto every row. The Europe PMC **website** sits behind a Cloudflare check and its robots file sends a content signal of `ai-train=no`; this Actor does not scrape the website. NCBI describes the PMC open-access article set as released for reuse under the article licences. **arXiv.** The API terms (info.arxiv.org) say metadata, including the abstract, is CC0, and that storing or serving the e-print itself is allowed only when the licence the submitter chose permits redistribution. They also say most e-prints use the arXiv non-exclusive licence, which does not. This Actor follows that line: non-exclusive and unknown licences do not get a body. HTML is fetched from `arxiv.org/html/`, which robots.txt allows, with the 15-second crawl delay. The API is called at most once every three seconds. arXiv is not the copyright holder of the papers. **OpenAlex.** The catalogue is CC0 ("free data, paid service"). Single-record lookups are free without a key. Search and cached-PDF downloads are metered by OpenAlex, so they run only with a key you provide, and they spend your allowance. This Actor does not ship a key of its own. The weekly watch does not call OpenAlex search. **Unpaywall.** The lookup is their free API (`api.unpaywall.org/v2/{doi}`). It requires your email in the query and asks callers not to use a shared or placeholder address. This Actor sends the address you type, at most one request a second, and does not store it. Only `url_for_pdf` is downloaded, and only when that location's licence allows redistribution. Publisher HTML is not scraped. **Your obligations.** You decide what you do with the text afterwards. NC means no commercial use. ND may not allow an adapted copy; the row says so. CC BY requires attribution, which the `citation` field provides. Do not use this for paywalled full text. If a rightsholder asks, the Actor will be taken down; use the Issues tab on this page. **Not affiliated** with arXiv, Europe PMC, PubMed, NCBI, OpenAlex, Unpaywall, Crossref, ScholarAPI, or any journal or publisher. Names are used only to say where the text came from. ### FAQ **Does it need a key?** No. Add an OpenAlex key only if you want OpenAlex keyword search or OpenAlex's cached PDF. The weekly watch and Unpaywall do not use that key. Unpaywall needs your email, not a key. **How do I watch for new papers?** Clear **Papers**, set **New papers this week**, put the keyword in **Search query**, leave **Maximum papers** at the number you are willing to pay for, and schedule the Actor weekly. Europe PMC and arXiv only. A quiet week is a successful run with no charge beyond the start fee. **Does it scrape Google Scholar?** No. Scholar is not a source. **Why was my arXiv paper skipped?** The HTML licence box was not a Creative Commons or CC0 licence. That is the usual arXiv licence, and the text stays on arXiv. You are not charged. **Why are page numbers empty?** XML and HTML are not paginated. Use `headings`. PDF rows have `pageStart` and `pageEnd`. **Why did a search return fewer papers than I asked for?** `maxPapers` caps the run, duplicates are removed, and papers without a redistributable full text are skipped rather than padded. **Is a failed paper billed?** No. Only `status: "processed"` is a Research paper event. **How do I report a problem?** Open an issue on this Actor with the run id and the paper id. ### Other Actors | Actor | What it does | |---|---| | [PDF to Markdown & RAG Chunks](https://apify.com/mmdincer/pdf-rag-extractor) | The same chunk shape for a PDF file you already have a link to | | [Image to Text & Scanned PDF OCR](https://apify.com/mmdincer/image-ocr-rag) | OCR when the PDF is a scan and has no text layer | | [GitHub Repo RAG](https://apify.com/mmdincer/github-repo-rag-extractor) | A repository as token-counted chunks, with the repo licence | # Changelog This Actor's version history is a separate document: https://apify.com/mmdincer/research-paper-rag/changelog.md # Actor input Schema ## `papers` (type: `array`): DOIs, PMIDs, PMCIDs, arXiv ids, OpenAlex work ids, or links to those records. One per line, up to 25. Example: PMC4792175 or arXiv:2405.13832 or 10.1038/sdata.2016.18. ## `query` (type: `string`): Optional, unless New papers this week is on. Finds open-access papers with full text. Europe PMC (PubMed OA) and the arXiv API are searched. OpenAlex keyword search runs only when you supply your own OpenAlex key, because that call is billed by OpenAlex. A weekly watch does not search OpenAlex. ## `sources` (type: `array`): Which official sources to ask. Europe PMC serves PubMed open-access full text. arXiv serves HTML when the paper's licence allows redistribution. OpenAlex supplies the licence, the ids and, when you add a key, its cached PDF. ## `maxPapers` (type: `integer`): Stop after this many papers, identifiers and search hits combined. A low number caps the cost of a first run. ## `datasetId` (type: `string`): Optional. An Apify dataset whose items contain doi, pmcid, pmid, arxivId, openAlexId, url or id. The dataset is opened read-only. ## `watchNewPapers` (type: `boolean`): Keyword watch. With a search query, Europe PMC and arXiv return papers first published in the last 7 days (Europe PMC FIRST_PDATE, arXiv submittedDate). Schedule the Actor once a week. Clear the Papers list, or those ids are read on every run. Papers this query already returned are skipped and not charged. arXiv HTML is opened so the licence box can be checked; the body is kept only when that licence allows redistribution. OpenAlex is not searched. You still pay the Research paper event only when full text is returned. ## `mode` (type: `string`): Paper and chunks writes one paper row plus its chunk rows. Chunks only writes chunk rows for papers that were read (failed papers still get a paper row so you can see why). Paper only skips chunking. ## `chunkSize` (type: `integer`): Token budget per chunk, counted with cl100k_base, the tokenizer OpenAI embedding and chat models use. 512 suits retrieval. Use 256 for narrow facts and 1024 for long arguments. ## `chunkOverlap` (type: `integer`): How many tokens from the end of one chunk are repeated at the start of the next. At most half the chunk size. ## `includeMarkdown` (type: `boolean`): Add the full markdown field on the paper row. Turn it off when you only want the chunks. ## `includeReferences` (type: `boolean`): Off by default. Reference lists are long and rarely useful in a retrieval index. Turn this on when you want the bibliography in the chunks. ## `maxChunksPerPaper` (type: `integer`): Safety cap on chunk rows for one paper. The paper row is marked truncated when the cap is hit. Chunks are not charged. ## `licenseFilter` (type: `string`): Any open licence returns CC BY, CC BY-SA, CC BY-NC, CC BY-ND, the NC-ND combinations, CC0 and public domain, and marks which ones allow commercial use or adaptation. Commercial use skips NC licences. The arXiv non-exclusive licence is never returned as full text. ## `openAlexApiKey` (type: `string`): Your own free OpenAlex key. Needed only for OpenAlex keyword search and for OpenAlex's cached PDF when Europe PMC and arXiv have no text. Those calls spend your OpenAlex daily allowance (about $0.001 per search and $0.01 per PDF). The key is not written to the dataset or the log. Single-record lookups work without a key. ## `contactEmail` (type: `string`): Your own email, sent only to the Unpaywall API when a DOI has no redistributable full text in Europe PMC or arXiv. Unpaywall requires it and rejects placeholder domains such as example.com. The address is not written to the dataset, the run summary or the log. Leave this empty to skip Unpaywall. Only a PDF whose location licence allows redistribution is downloaded. ## Actor input object example ```json { "papers": [ "PMC4792175", "arXiv:2405.13832" ], "sources": [ "europepmc", "openalex", "arxiv" ], "maxPapers": 5, "watchNewPapers": false, "mode": "both", "chunkSize": 512, "chunkOverlap": 64, "includeMarkdown": true, "includeReferences": false, "maxChunksPerPaper": 400, "licenseFilter": "any-open" } ``` # Actor output Schema ## `documents` (type: `string`): One row per paper with its status, citation, licence and, when the licence allows, the Markdown. Filter on recordType document. ## `chunks` (type: `string`): One row per token-counted chunk with the text, the headings and the citation. Filter on recordType chunk. This is the view to send to an embedding model. ## `summary` (type: `string`): How many papers were processed, skipped or failed, how many chunks were written, and why the run stopped. A weekly watch also records the date window and how many hits were already returned. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "papers": [ "PMC4792175", "arXiv:2405.13832" ], "sources": [ "europepmc", "openalex", "arxiv" ], "maxPapers": 5, "watchNewPapers": false, "mode": "both", "chunkSize": 512, "chunkOverlap": 64, "includeMarkdown": true, "includeReferences": false, "maxChunksPerPaper": 400, "licenseFilter": "any-open" }; // Run the Actor and wait for it to finish const run = await client.actor("mmdincer/research-paper-rag").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "papers": [ "PMC4792175", "arXiv:2405.13832", ], "sources": [ "europepmc", "openalex", "arxiv", ], "maxPapers": 5, "watchNewPapers": False, "mode": "both", "chunkSize": 512, "chunkOverlap": 64, "includeMarkdown": True, "includeReferences": False, "maxChunksPerPaper": 400, "licenseFilter": "any-open", } # Run the Actor and wait for it to finish run = client.actor("mmdincer/research-paper-rag").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "papers": [ "PMC4792175", "arXiv:2405.13832" ], "sources": [ "europepmc", "openalex", "arxiv" ], "maxPapers": 5, "watchNewPapers": false, "mode": "both", "chunkSize": 512, "chunkOverlap": 64, "includeMarkdown": true, "includeReferences": false, "maxChunksPerPaper": 400, "licenseFilter": "any-open" }' | apify call mmdincer/research-paper-rag --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,mmdincer/research-paper-rag" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/pXfwbRnEJbZGbY1fv/builds/XvMxxkDfkXcOO9HIF/openapi.json