arXiv Paper-to-JSON scraper

arXiv Papers Scraper

devilscrapes/arxiv-papers-scraper

Search arXiv by query, category, or author and get structured paper metadata — title, authors, abstract, primary category, DOI, PDF URL, submitted and updated timestamps. We handle pagination, retries, and rate-limit pacing so you get clean typed rows ready for a research pipeline.

Scrape Zenodo.org (CERN open research repository) for records, datasets, and software. Four modes: search with type/access filters, record details by DOI/ID, community browse, recent submissions. Extracts titles, authors, DOIs, files, stats. Uses official API. No auth, 60 req/min.

Scrape HuggingFace Hub: models, datasets, spaces. 30+ fields per record, trending filters, author profiles, parsed tags, web enrichment for emails & websites.

📄 arXiv Papers Monitor

skootle/arxiv-papers

Pull new AI / ML / CS / physics / math papers from arXiv as they land via the official arXiv API. Title, abstract, authors, PDF link, DOI, and LLM-ready summary card per paper. For ML researchers, AI agents, and journalists. Export, run via API, schedule, or integrate with other tools.

Search arXiv and get clean structured JSON for each paper: title, authors, abstract, categories, DOI, PDF link, and dates. Built for research, datasets, and AI pipelines.

Scrape GBIF (Global Biodiversity Information Facility) for species and occurrences. Modes: species search, occurrence records, dataset browse, country filters. Extracts scientific names, lat/long, dates, publishers, images, license. Official REST API, no auth.

Academic Paper Scraper

constant_quadruped/academic-paper-scraper

Search arXiv and PubMed in one request. Returns unified paper data: titles, authors, abstracts, DOIs, and PDF links. Filter by keywords, authors, categories, and date range. Built-in rate limiting and cross-source deduplication. Export to JSON, CSV, or Excel.

ArXiv Research Paper Scraper

datapilot/arxiv-research-paper-scraper

arXiv Research Paper Scraper retrieves academic paper metadata from the arXiv API based on a keyword. It extracts titles, abstracts, authors with affiliations, DOI, categories, submission dates, and PDF links. Supports proxy usage and outputs structured JSON results for research and data analysis.

Scrape Kaggle datasets marketplace. Modes: search by keyword/tag, dataset details (owner, license, file list, size, votes, downloads), trending, and user profiles. Extracts titles, descriptions, updated dates, usability scores. Ideal for ML dataset discovery and competitive landscape research.

arXiv Scraper for RAG: Papers as Chunked JSON

devanshlive/arxiv-rag-extractor

Scrape arXiv papers by date and category. Strips LaTeX and returns RAG-ready JSON with tokenizer-aware chunks (cl100k_base, 512/50). Drop-in for LangChain, LlamaIndex, Qdrant, Pinecone, Weaviate, pgvector, Chroma. Skip GROBID / Nougat / pandoc. $0.015 per paper.