Academic Corpus Harvester

Harvest academic paper metadata from arXiv, enrich with OpenAlex citation/concept data, and optionally extract full-text from open-access PDFs. Outputs clean structured JSON or RAG-ready Markdown chunks.

Why this Actor?

Building AI/ML applications requires high-quality academic data — papers, citations, and concepts. This Actor gives you programmatic access to the arXiv + OpenAlex ecosystem:

• Build RAG datasets for scientific question answering or literature review
• Create training data for fine-tuning domain-specific language models
• Track citation impact across research fields
• Analyze research trends by concept, category, or time period

Unlike general web scrapers, this Actor speaks the native APIs of arXiv and OpenAlex, understands the data schema, and handles pagination, rate limits, and enrichment automatically.

Input

Field	Type	Default	Description
query	string	required	Search terms or arXiv category code (e.g. transformer attention or cs.LG)
dateFrom	string	—	ISO date lower bound (e.g. 2024-01-01)
dateTo	string	—	ISO date upper bound (e.g. 2024-12-31)
maxResults	integer	100	Max papers to return (hard cap: 1000)
enrichCitations	boolean	true	Look up citation count and concepts from OpenAlex
includeFullText	boolean	false	Download and extract text from open-access PDFs (increases compute)
outputFormat	enum	json	json for structured records, markdown-chunks for RAG-ready chunks
chunkSize	integer	500	Target words per chunk (only used with markdown-chunks output)

Output

Each paper record includes:

id              — arXiv ID
title           — paper title
authors         — list of author names
abstract        — paper abstract
publishedDate   — ISO publication date
source          — "arxiv"
sourceUrl       — link to arXiv abstract page
pdfUrl          — link to PDF (may be null)
categories      — arXiv category tags
citationCount   — OpenAlex citation count (null if enrichment disabled)
concepts        — OpenAlex topic tags (null if enrichment disabled)
fullText        — extracted PDF text (null if full-text disabled)
markdownChunks  — chunks for RAG pipelines (null if json output)

Pricing (Pay-Per-Event)

Event	Fired	Suggested Price
metadata-record	Once per paper metadata record	$0.001–0.002
citation-enrichment	Once per paper enriched via OpenAlex	$0.003–0.005
full-text-extracted	Once per paper with PDF text extracted	$0.01–0.02

Data Sources

• arXiv — Open-access repository of 2M+ scholarly articles. No API key required.
• OpenAlex — Open catalog of research works, authors, and concepts. No API key required.

Use Cases

• RAG over Scientific Literature — Chunk papers into markdown-chunks and feed them into a vector database
• Citation Network Analysis — Enrich papers with OpenAlex citation counts to identify influential works
• Trend Discovery — Search by category + date range to track emerging research directions
• Dataset Building — Build custom corpora for fine-tuning domain-specific LLMs

Local Development

# Enter the Nix devShell
nix develop

# Install dependencies
uv sync

# Run tests
uv run pytest

# Test locally with Apify CLI
apify run

Technical Notes

• arXiv API returns Atom XML (parsed via lxml)
• OpenAlex REST API queried by arXiv ID; gracefully handles misses
• PDF extraction uses pypdf with pdfplumber fallback for poor-quality extractions
• Chunking uses word-count-based splitting with 10% overlap for RAG readiness