Research Papers to Markdown for AI Agents

Search academic papers by topic, or look them up by DOI, arXiv ID, PubMed ID (PMID), PubMed Central ID (PMCID) or OpenAlex ID, and get one clean item per paper with its metadata and, when its open license allows it, the abstract and full text as markdown for AI agents. Each item carries the title, authors, year, venue, citations, open access status, links, IDs and the attribution the license asks for: for literature reviews, research agents and RAG over open-access papers.

• Metadata comes from OpenAlex (CC0), which covers Crossref, PubMed and arXiv records.
• Abstract and full text only for papers with an open license that allows reuse: CC BY, CC BY-SA, CC BY-ND, CC0 or public domain.
• Charged per paper delivered, and once more per paper delivered with its full text; IDs that are not found are free.

More: Research Papers to Markdown on frameskim.dev

What you get

{
  "id": "https://openalex.org/W4389519118",
  "doi": "https://doi.org/10.18653/v1/2023.emnlp-main.495",
  "title": "Active Retrieval Augmented Generation",
  "authors": ["Zhengbao Jiang", "Frank F. Xu", "Luyu Gao", "Zhiqing Sun", "Qian Liu"],
  "year": 2023,
  "publicationDate": "2023-01-01",
  "venue": "Conference on Empirical Methods in Natural Language Processing (EMNLP)",
  "type": "conference-paper",
  "citedByCount": 497,
  "isOpenAccess": true,
  "oaStatus": "hybrid",
  "license": "cc-by",
  "landingPageUrl": "https://doi.org/10.18653/v1/2023.emnlp-main.495",
  "pdfUrl": "https://aclanthology.org/2023.emnlp-main.495.pdf",
  "ids": {"openalex": "https://openalex.org/W4389519118", "doi": "https://doi.org/10.18653/v1/2023.emnlp-main.495", "pmid": null, "pmcid": null, "arxiv": null},
  "abstract": "Despite the remarkable ability of large language models...",
  "fullText": "# Active Retrieval Augmented Generation\n\n...",
  "fullTextUrl": "https://aclanthology.org/2023.emnlp-main.495.pdf",
  "fullTextError": null,
  "truncated": false,
  "attribution": "\"Active Retrieval Augmented Generation\" by Zhengbao Jiang, Frank F. Xu, Luyu Gao et al., Conference on Empirical Methods in Natural Language Processing (EMNLP) (2023). License: CC BY (https://creativecommons.org/licenses/by/), for the copy at https://doi.org/10.18653/v1/2023.emnlp-main.495. https://doi.org/10.18653/v1/2023.emnlp-main.495. Metadata: OpenAlex (CC0)."
}

• A missing field is null. license is the license of the open copy the text comes from, as OpenAlex reports it.
• Without an open license that allows reuse, abstract and fullText are null and fullTextError says why; the paper is still delivered with its metadata.
• Full text is read only from the open copy's PDF, and only on sites whose terms allow automated downloads (for example ACL Anthology, PLOS, eLife, Zenodo). For any other site, including arXiv and PubMed Central, the item gives the link in fullTextUrl and says why in fullTextError. Scanned PDFs (images only) are not read.
• An ID that could not be delivered has input and error, with the reason, and is not charged.
• The run's status message says how many papers were delivered, how many with full text, how many failed and why the run stopped early, if it did.
• If nothing is delivered, the run still ends as succeeded: each paper that failed, or a search that found nothing or could not be run, is in the dataset with its input and error, the status message gives the first reason, and nothing was charged.

Input

Field	What it does
query	What to search: topic, title words or author, up to 300 characters.
ids	From 1 to 100 IDs: DOIs, arXiv IDs, PMIDs, PMCIDs or OpenAlex IDs. A repeated ID is read once. When given, the search is not run.
maxResults	From 1 to 100 (default 10): how many papers the search returns.
fullText	Also read the full text when the license allows it (on by default).
openAccessOnly	Search only papers that have a free copy.
fromYear	Search only papers published in or after this year.

Limits

• PDFs up to 5 MB, 15 s per download attempt and 30 s to convert each; 3 papers at a time. A download that runs out of time, cannot connect or gets a server error is tried once more, except on sites that ask for a pause between downloads; if that fails too, the paper comes back without its text and the text is not charged.
• One PDF per run from sites that ask crawlers to wait between requests.
• Set a maximum charge per run to cap spending: the run stops before a paper that would pass it.

Pricing

This actor bills two events, nothing else:

• paper: one per paper delivered with its metadata.
• full-text: one more per paper delivered with its full text as markdown.

IDs that are not found, papers that cannot be read and full texts that cannot be read are not charged.

Site owners

PDFs are read as the FrameskimFetch bot, which respects robots.txt on every request. To keep your PDFs out, disallow FrameskimFetch in your robots.txt (the other Frameskim actors that fetch pages also honor that user agent, so the rule stops them too). Rights holders can also write to support@frameskim.dev. Neither a robots.txt rule nor a request removes text from datasets that users already hold.

Terms and privacy

Research Papers to Markdown for AI Agents is operated by Emttn LLC and is not affiliated with, endorsed by or sponsored by OpenAlex or OurResearch, arXiv, the U.S. National Library of Medicine or PubMed Central, or the publishers, repositories and authors of the papers it returns. You are responsible for following each paper's license when you reuse its metadata, abstract or text; the license shown comes from OpenAlex and may be wrong, so check it on the paper's own page. Items carry the authors' names, public bibliographic data, so you can give the attribution the license asks for. Terms: https://frameskim.dev/terms. Privacy: https://frameskim.dev/privacy. Questions: support@frameskim.dev.

Not affiliated with the International DOI Foundation; DOI is named here only as the identifier you look papers up by.

Support

Every error says what was wrong. For anything else, see https://frameskim.dev/terms.