Inspire Hep Papers Scraper

INSPIRE-HEP Scraper: Physics Papers and Authors

Explore and extract high-energy physics research literature and author profiles directly from INSPIRE-HEP. BigiByte's INSPIRE-HEP Scraper pulls citations, author names, arXiv IDs, and publication metadata into clean, structured datasets—saving you from manual database queries and pagination.

---

What Does This Scraper Do?

INSPIRE-HEP is the central digital library for high-energy physics literature, connecting researchers with global preprints, journal articles, and author profiles. This tool helps researchers, academic institutions, and science writers quickly gather comprehensive literature and author records into clean tabular formats.

Key Use Cases

• Literature Reviews: Pull papers for specific research topics (e.g., Higgs bosons, quantum field theory, cosmology) with citation counts and arXiv identifiers in a single dataset.
• Author Profiles & Collaborations: Export matching author records, including names and institutional histories.
• Citation Tracking: Analyze citation counts stored directly on each paper record to evaluate impact and follow research trajectories.
• Research Operations: Export capped batches of structured records straight into your data notebooks or local storage.
• Scheduled Refreshes: Automate periodic data collection runs to keep your reference lists current without manual intervention.

---

Input Parameters

Parameter	Type	Default	Description
mode	String	literature	Choose literature for research papers or authors for author records.
query	String	higgs	The search term or query string to match against records.
type	String	literature	Optional alias for mode.
maxItems	Integer	100	Maximum number of records to collect (capped at 1,000).
timeoutSecs	Integer	300	Overall execution timeout in seconds.
requestTimeoutSecs	Integer	30	Timeout per network request in seconds.

Example Input

{
    "mode": "literature",
    "query": "higgs",
    "maxItems": 100,
    "timeoutSecs": 300,
    "requestTimeoutSecs": 30
}

---

Extracted Output Data

Each saved dataset row contains complete record metadata along with simplified top-level fields for convenience:

{
    "links": {
        "json": "https://inspirehep.net/api/literature/2181837?format=json",
        "citations": "https://inspirehep.net/api/literature/?q=refersto%3Arecid%3A2181837"
    },
    "created": "2022-11-15T03:07:31.445632+00:00",
    "updated": "2023-06-14T16:26:19.477731+00:00",
    "metadata": {
        "citation_count": 1,
        "control_number": 2181837,
        "titles": [
            {
                "source": "arXiv",
                "title": "Study of $HZZ$ anomalous couplings by angular differential cross sections"
            }
        ],
        "arxiv_eprints": [
            {
                "categories": ["hep-ph"],
                "value": "2211.07478"
            }
        ]
    },
    "id": "2181837",
    "paperTitle": "Study of $HZZ$ anomalous couplings by angular differential cross sections",
    "citationCount": 1,
    "arxivId": "2211.07478",
    "scrapedAt": "2026-10-12T00:00:00+00:00"
}

Output Field Reference

Field	Type	Description
links	Object	Direct links object associated with the record.
created	String	Record creation timestamp.
updated	String	Last update timestamp.
metadata	Object	Detailed record metadata (sensitive fields like email addresses are automatically excluded).
id	String	Unique INSPIRE record identifier.
paperTitle	String	Primary paper title (literature records only).
citationCount	Integer	Total citation count (when available).
arxivId	String	Associated arXiv preprint identifier (when available).
authorName	String	Author name (author records only).
scrapedAt	String	UTC timestamp indicating when the item was collected.
error	String	Populated only if a specific request encounters an issue, ensuring the task continues running smoothly.

---

How Data Collection Works

• Optimized Pagination: Pages through results sequentially while respecting platform rate limits and item windows.
• Automatic Deduplication: Duplicate record IDs are automatically filtered out to keep your dataset clean and unique.
• Resilient Retries: Built-in network error handling manages transient rate limits or server timeouts smoothly without crashing the entire run.
• Flexible Export: Download your collected research datasets anytime in JSON, CSV, or Excel formats.

---

Related Tools & Custom Development

If you plan to map, cross-reference, or join physics literature records with external institutional datasets, quick CSV cleanups or bibliographic mapping passes are standard next steps. BigiByte builds custom data scrapers, research feeds, and automated monitoring pipelines for academic institutions and scientific repositories worldwide.

Need custom scheduled monitoring, specialized database integration, or tailored data pipelines? BigiByte builds custom scrapers and APIs tailored to your project: https://bigibyte.com

---

Frequently Asked Questions

How many records can I extract per run?

maxItems defaults to 100 records and can be adjusted up to a hard cap of 1,000 successful items per run.

Can I export my data directly to CSV or Excel?

Yes! Once a task completes, you can export the full dataset directly to JSON, CSV, or Excel format.

Do author records include email addresses?

No. To maintain privacy and data compliance, any fields resembling email addresses are automatically stripped before records are saved.

---

Trademark Notice & Disclaimer

Trademark Notice: INSPIRE-HEP and related names, logos, and trademarks belong to their respective owners. Their use in this documentation is solely to identify the source of data and does not imply affiliation, sponsorship, endorsement, or partnership.

Disclaimer: This tool is an independent solution developed by BigiByte and is not officially affiliated with, endorsed by, or sponsored by INSPIRE-HEP. All data is collected from publicly accessible records. Users are responsible for adhering to the source's terms of service, rate limits, and licensing requirements.

---

Is Web Scraping Legal?

This tool collects publicly available physics literature and author metadata from INSPIRE-HEP. While extracting public records is standard practice, consider the following best practices:

• Personal Data: Author names and professional profiles constitute public research data. Ensure compliance with applicable privacy regulations (such as GDPR or CCPA) if storing or processing personal information.
• Terms & Rate Limits: Respect platform guidelines and rate limits (such as managing request pacing and adhering to metadata licensing terms like CC0 where applicable). Bulk harvesting of restricted metadata fields (such as email addresses) is strictly avoided.
• Legal Context: This information provides general context and does not constitute legal advice.

To learn more, read Apify's guide: Is web scraping legal?