arXiv Scraper — Search & Export Paper Metadata
Pricing
from $1.32 / 1,000 result emitteds
Search arXiv by query, category, or author and export structured paper metadata — title, authors, abstract, primary category, DOI, PDF URL, submitted and updated timestamps — to JSON or CSV. An arXiv API wrapper that handles pagination, retries, and rate-limit pacing for your pipeline.
_fetch_page in
src/scraper.py. The arXiv Atom API (export.arxiv.org) can return
HTTP 503 during weekly maintenance windows or 429 under rate-limiting;
it also occasionally returns an HTML maintenance page with HTTP 200,
causing ET.ParseError. All three cases are now retried up to 5
attempts (backoff: 2 s → 4 s → 8 s → 16 s → 30 s, capped).
Retry-After headers are honoured. Non-retryable 4xx responses
(e.g. 400 bad query) still raise immediately. After all attempts
exhaust, the run fails loud with a RuntimeError — never silently
returning an empty dataset.RETRY_STATUS_CODES, BACKOFF_BASE_S, BACKOFF_CAP_S,
MAX_ATTEMPTS, RETRY_AFTER_HEADER._backoff_delay, _parse_atom (pure, unit-testable).tests/test_scraper_retry.py covering: 503×2 then success,
persistent 503, HTML-maintenance-page 200 retry, non-retryable 400,
Retry-After header, backoff growth/cap, and _parse_atom directly.