Inspire Hep Scraper

Search inspire hep literature and export normalized paper records from the official INSPIRE HEP API. Build recurring research datasets with titles, abstracts, authors, affiliations, collaborations, citation counts, subjects, publication dates, identifiers, full-text links, and canonical source URLs.

The Actor is designed for physicists, librarians, scientometric analysts, and data teams that need structured INSPIRE records without maintaining API pagination and normalization code.

What does Inspire Hep Scraper do?

The Actor turns an INSPIRE literature search into integration-ready dataset rows. You can:

• search by free text or native INSPIRE query syntax;
• filter by author, collaboration, subject, and inclusive date range;
• sort by recency, citation count, or relevance;
• export up to 10,000 records per run;
• retain normalized fields, authoritative source links, and BibTeX/LaTeX citation links;
• schedule the same query and compare datasets downstream.

It reads the public official JSON API directly. It does not scrape rendered pages, require login credentials, or download full-text documents.

Who is it for?

High-energy-physics researchers

Create a compact evidence set for a topic, collaborator, or experiment before a literature review.

Research librarians

Export stable identifiers, publication references, author affiliations, and source provenance into catalog or discovery workflows.

Scientometric analysts

Collect citation counts, citation links, subjects, collaborations, DOI values, and arXiv identifiers for reproducible analysis.

Data and automation teams

Run a date-bounded query on a schedule, deliver results to a warehouse, and detect new or updated records with your preferred integration.

Why use this Actor?

• Official source: results come from inspirehep.net/api/literature.
• Normalized rows: nested API metadata becomes predictable dataset fields.
• Research context: citations, collaborations, subjects, author IDs, affiliations, DOIs, and arXiv IDs remain together.
• Bounded execution: maxItems, request timeout, pagination, and retry limits prevent uncontrolled runs.
• Transparent provenance: every row includes canonical page, record API, citation-query, source, query, rank, and retrieval time.
• No account required: supported public searches use the anonymous INSPIRE HEP API.

What data can I extract?

Field	Meaning
inspireId	Stable literature record ID
controlNumber	INSPIRE control number when supplied
title	Primary paper title
abstract	Primary abstract, or null when unavailable
authors	Structured names, INSPIRE author IDs, ORCID values, and affiliations
authorNames	Flat author-name list for spreadsheets
affiliations	Unique normalized and raw affiliations across authors
collaborations	Named collaborations such as ATLAS or CMS
citationCount	Current citation count reported by INSPIRE
citationCountWithoutSelfCitations	Citation count excluding self-citations when available
dois	DOI values
arxivIds	Linked arXiv identifiers
texkeys	INSPIRE TeX citation keys
documentTypes	Record document types
subjects	INSPIRE subject classifications
publicationInfo	Journal, volume, year, page, and article reference data
earliestDate	Earliest known literature date
preprintDate	Preprint date
createdAt	Record creation timestamp
updatedAt	Record update timestamp
fullTextUrls	Public document links exposed by INSPIRE
canonicalUrl	Canonical human-readable literature page
apiUrl	Official record API URL
citationsUrl	Official API query for citing papers
bibtexUrl	Official BibTeX representation link, or null
latexEuUrl	Official LaTeX EU representation link, or null
latexUsUrl	Official LaTeX US representation link, or null
query	Effective query sent to INSPIRE
rank	One-based export rank
totalResultsForQuery	Total matches reported by INSPIRE
source	INSPIRE HEP
retrievedAt	UTC retrieval timestamp

Field availability follows the source record. An empty author, DOI, abstract, collaboration, or full-text list is valid when INSPIRE does not expose that metadata.

How to use Inspire Hep Scraper

1. Open the Actor in Apify Console.
2. Enter at least one query or structured filter.
3. Choose a sort order.
4. Set the maximum number of papers.
5. Click Start.
6. Open the Dataset tab to review, download, or integrate the rows.

A useful first run is:

{
  "collaboration": "ATLAS",
  "sort": "mostrecent",
  "maxItems": 10
}

Input parameters

Parameter	Type	Default	Description
query	string	—	Free text or native INSPIRE literature query syntax
author	string	—	Author name such as Witten, Edward
collaboration	string	—	Collaboration such as ATLAS, CMS, or LIGO
subject	string	—	INSPIRE subject such as Theory-HEP
dateFrom	string	—	Inclusive lower date in YYYY, YYYY-MM, or YYYY-MM-DD
dateTo	string	—	Inclusive upper date in the same format
sort	string	mostrecent	mostrecent, mostcited, or relevance
maxItems	integer	100	Maximum saved papers, from 1 to 10,000
maxRequestRetries	integer	3	Retries for timeout, HTTP 429, and temporary server failures, from 0 to 5

All supplied search clauses are combined with and. Text values are trimmed; structured author/collaboration/subject values become quoted INSPIRE field queries, not local exact whole-field comparisons. INSPIRE controls token, phrase and case matching: for example, ATLAS can also match a collaboration such as ATLAS ITk Strips. For example, subject: Theory-HEP plus a 2025 range searches only matching records in that period. Dates must be real calendar dates, including leap-year rules. Invalid dates and wrong input types fail before charging or source requests. Lower year/month bounds begin on the first day; upper bounds include the last day. Mixed precision is supported: dateFrom: "2025-01-01" with dateTo: "2025" is valid. Original date precision is preserved in INSPIRE's native date query.

Search examples

Search a research topic

{
  "query": "quantum gravity",
  "sort": "mostcited",
  "maxItems": 25
}

Search an author

{
  "author": "Witten, Edward",
  "sort": "mostcited",
  "maxItems": 50
}

Export collaboration literature

{
  "collaboration": "ATLAS",
  "sort": "mostrecent",
  "maxItems": 100
}

Build a date-bounded subject dataset

{
  "subject": "Theory-HEP",
  "dateFrom": "2025-01-01",
  "dateTo": "2025-12-31",
  "sort": "mostrecent",
  "maxItems": 500
}

Use native INSPIRE syntax

{
  "query": "refersto:recid:451647",
  "sort": "mostcited",
  "maxItems": 100
}

Native query syntax is passed to INSPIRE inside parentheses and can be combined with the structured filters. Invalid syntax returns a failed run with the API error instead of a misleading empty dataset.

Retrieve bibliography links for a known paper

{
  "query": "recid:451647",
  "maxItems": 1
}

Open the Bibliography links dataset view for citation keys, identifiers and official format links. For record 451647, bibtexUrl is https://inspirehep.net/api/literature/451647?format=bibtex; the LaTeX links use format=latex-eu and format=latex-us. Missing or invalid source links are null. These fields are URLs, not exported BibTeX/LaTeX files. The Actor copies links already present in search hits without extra per-paper requests, downloads, or charge events. The existing paper overview remains available.

Output example

This abbreviated shape reflects real current Actor output:

{
  "inspireId": "101338",
  "controlNumber": 101338,
  "title": "Particle Creation by Black Holes",
  "abstract": "In the quantum theory of fields in curved space-time...",
  "authorNames": ["Hawking, S.W."],
  "affiliations": ["Cambridge U., DAMTP"],
  "collaborations": [],
  "citationCount": 13978,
  "dois": ["10.1007/BF02345020"],
  "arxivIds": [],
  "documentTypes": ["article"],
  "subjects": ["Gravitation and Cosmology"],
  "canonicalUrl": "https://inspirehep.net/literature/101338",
  "apiUrl": "https://inspirehep.net/api/literature/101338",
  "query": "(quantum gravity)",
  "rank": 1,
  "source": "INSPIRE HEP"
}

The full dataset also contains structured authors, publication references, source timestamps, full-text URLs, citation URL, total match count, and retrieval timestamp.

How much does it cost to export INSPIRE HEP papers?

Pay-per-event pricing contains a $0.005 run start and one paper event for every normalized dataset row. On the BRONZE plan, a paper costs $0.003872.

Papers	BRONZE calculation
1	0.005 + 1 × 0.003872 = 0.008872 USD
10	0.005 + 10 × 0.003872 = 0.043720 USD
100	0.005 + 100 × 0.003872 = 0.392200 USD
1,000	0.005 + 1,000 × 0.003872 = 3.877000 USD

The formula is $0.005 + paper count × plan paper price. No paper event is charged for a duplicate, rejected, or empty record. Rows are limited to the remaining paper-event budget, and only successfully charged papers are saved. A storage failure after charging fails the run; charging and storage are not a single atomic transaction. Apify platform usage may also apply according to your Apify plan.

Recurring literature monitoring

The Actor returns snapshots, not alerts or change records. To build a monitor:

1. choose a subject, author, collaboration, or native query;
2. add a recent date boundary;
3. save the Actor input as an Apify Task;
4. schedule that Task;
5. send each dataset to a database, webhook workflow, or cloud storage;
6. compare stable inspireId values and updatedAt timestamps downstream.

This keeps monitoring logic explicit and avoids claiming that every run contains only newly changed records.

Dataset export and integrations

The default dataset works with Apify's standard export formats, including JSON, CSV, XML, Excel, and RSS where supported. Common workflows include:

• Google Sheets for a review queue;
• BigQuery or Snowflake for scientometric analysis;
• Make or Zapier for notifications;
• webhooks for ingestion after a successful run;
• Python notebooks for citation-network analysis;
• an LLM retrieval pipeline that stores titles and abstracts with source links.

Use inspireId as the stable deduplication key. Use updatedAt to detect source metadata changes.

Run with the Apify API

Replace <APIFY_TOKEN> with your Apify API token.

cURL

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~inspire-hep-literature-search/runs?token=<APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"collaboration":"ATLAS","sort":"mostrecent","maxItems":25}'

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/inspire-hep-literature-search').call({
  collaboration: 'ATLAS',
  sort: 'mostrecent',
  maxItems: 25,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/inspire-hep-literature-search').call(run_input={
    'subject': 'Theory-HEP',
    'dateFrom': '2025-01-01',
    'dateTo': '2025-12-31',
    'maxItems': 100,
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)

Use with MCP and AI assistants

Add the Apify MCP server in Claude Code:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/inspire-hep-literature-search"

Claude Desktop setup

Add this HTTP MCP server in Claude Desktop's connector settings.

Cursor setup

Add the same HTTP endpoint in Cursor's MCP settings.

VS Code setup

Add the endpoint to your VS Code MCP server configuration.

Claude Desktop, Cursor, and VS Code can use this equivalent JSON:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/inspire-hep-literature-search"
    }
  }
}

Example prompts:

• “Export the 20 most-cited INSPIRE HEP papers matching quantum gravity.”
• “Find 25 recent ATLAS collaboration records and summarize the citation counts.”
• “Create a 2025 Theory-HEP dataset with DOI and arXiv identifiers.”

Follow one run to bounded results

1. Discover a stable subset. Keep the Actor-specific URL above. Selecting this Actor also automatically adds get-actor-run, get-dataset-items, get-key-value-store-record, and abort-actor-run; it does not expose exactly one tool. Use scoped tools/list to discover actual tool names and argument schemas, then select only the needed Actor/run/data tools. Discovery is read-only, not authorization to execute or abort.
2. Start once and retain the run. For an authorized invocation, save the returned run ID, status and storage IDs; these are metadata, not literature rows. call-actor waits at most waitSecs (0–45 seconds, default 30), so a response may be nonterminal. Follow that same run with get-actor-run, for example after 2, 4 and 8 seconds, capped at 10-second intervals, with a total 120-second consumer deadline. Set the client request timeout above the chosen server wait plus transport margin, within that deadline. After a client timeout, recover/check the known run instead of starting another run to poll. If its ID is unknown, report uncertainty rather than blindly rerunning. At the deadline, report pending with the run ID and stop polling. Report terminal failures honestly; available rows from a failed run are partial, not a successful full export.
3. Fetch rows within a context budget. After success, use get-dataset-items with the returned dataset ID (defaultDatasetId in the run API), explicit limit, offset and needed fields, using the discovered schema. For example, request untransformed JSON in pages of 20 source rows from offset 0, selecting inspireId, title, citationCount, canonicalUrl and bibtexUrl; admit at most 100 rows and 64 KiB of serialized UTF-8 result content across all pages to model context, whichever comes first. Track source offsets and pagination metadata, not transformed row counts: filtering or unwinding can change returned counts. Process pages host-side before model injection and enforce the byte budget even for an oversized row/page: omit or summarize with disclosure. If your client cannot intercept oversized responses, do not claim a hard byte guarantee or request unbounded results. Stop at exhaustion or budget and report returned rows, continuation offset and partial/truncated status when incomplete. Keep complete exports in Apify storage or downstream processing outside model context. This consumer context budget does not reduce the Actor's collection scope or output fields.
4. Qualify context claims. Tool-discovery transport bytes are not necessarily model-visible context. Discovery caching, progressive loading and prompt caching depend on the client and negotiated capabilities; there is no universal token-savings guarantee. Numeric savings claims require a measured client/version, capabilities/cache mode, model/tokenizer, visible tool/result subset and baseline method.

These backoff, deadline and context-budget values are consumer examples, not platform defaults. Always verify generated scientific conclusions against the source records.

Reliability and retry behavior

Each API request has a 30-second connection/response-header timeout and a bounded 90-second streamed-body timeout for large collaboration pages. The default overall run timeout remains 300 seconds; very large exports can require a larger user-selected timeout or narrower queries. The Actor retries network failures and temporary HTTP 5xx responses with bounded exponential backoff and jitter. HTTP 429 waits at least five seconds, honoring valid Retry-After seconds or HTTP dates up to 30 seconds. A longer server delay fails visibly with a retry-later message instead of retrying too soon. Retry counts are finite per page; prolonged outages can still reach the run timeout. Invalid queries, other deterministic HTTP 4xx responses, and malformed or non-JSON responses fail without blind retries. Error diagnostics do not include response bodies or effective queries.

Pagination uses a stable page size of up to 250 records, with streamed processing to keep large collaboration author lists memory-bounded at the 256 MiB default. Only consumed metadata fields are requested; authors and existing output fields are retained. Pages are validated and processed through temporary disk spools rather than buffering the whole export in memory. Responses up to 8 MiB use native JSON parsing; larger pages stream one hit at a time. A 128 MiB per-page transfer safety bound fails visibly rather than exhausting disk/memory; narrow a pathological query if this bound is reached. Charging remains once per nonempty source page, not per retry. The Actor deduplicates by inspireId, including within each page, and stops at maxItems, the end of source results, the remaining event budget, or INSPIRE's first 10,000 search positions. Invalid/duplicate rows never extend the source window (at size 250, page 41 is never requested). Page and final logs report cumulative requests, retries, rate limits, pages, received hits, saved rows, invalid skipped rows, duplicates, source total, downloaded response bytes, source/normalization/charging/storage timings, storage-request counts and stop reason (max_items, source_end, source_window, or budget). A capped or budget-limited export is not a claim of completeness; received and saved counts can differ.

If a transient outage continues beyond maxRequestRetries, the run fails so scheduled workflows can detect the problem.

Limits and data interpretation

• INSPIRE controls search matching, ranking, record coverage, and citation metrics.
• Citation counts can change as INSPIRE updates its graph.
• Some records have no abstract, authors, affiliation, DOI, arXiv ID, or full-text URL.
• fullTextUrls are links only; this Actor does not download PDFs.
• A broad query can match many records; maxItems limits unique saved rows, not all matches. INSPIRE exposes only the first 10,000 search positions; deduplication, invalid records, budget and runtime can reduce the export. Use narrower date/topic queries for larger collections.
• Date filters use INSPIRE's literature date query field.
• The Actor does not provide jobs, conferences, profiles, or private data.
• Scheduled comparisons and alerts must be implemented with Apify Tasks or downstream tooling.

Diagnostics and privacy

Failed operations send sanitized diagnostic input, exceptions, and Actor/build/run IDs to our private GlitchTip service for repair. Secret fields and URL query strings are removed; search queries, author names and structured filter values are also redacted from diagnostics. Reports are retained for 30 days. Successful datasets still contain the documented research query and public literature metadata, including public author names, affiliations and identifiers. Dataset/input/log retention follows your Apify storage settings; delete runs and their storages in Console or through the Apify API when no longer needed. Temporary page/record spools are deleted after consumption or handled errors and disappear with the run container after forced termination. No cross-run cache or external AI provider is used; output is normalized source metadata, not AI-generated scientific content.

Legality and responsible use

INSPIRE HEP is a public scholarly information service. This independently operated Actor is not affiliated with or endorsed by INSPIRE HEP or CERN. It uses the standard Apify Store terms and no extra service fee or paid external data dependency. Use the exported metadata in accordance with INSPIRE's terms, applicable database rights, publisher rights, and your organization's research policies.

Do not treat an accessible full-text link as permission to redistribute copyrighted documents. Do not use affiliation or author metadata for harassment, discrimination, or unsolicited personal targeting. Keep canonical source links so users can verify context and corrections.

Support

For a problem with this Actor, open an issue through its Apify Store page and include the run link and a non-sensitive description of the expected result. Do not post tokens or private research queries publicly.

Troubleshooting

The run says I must provide a filter

At least one of query, author, collaboration, subject, dateFrom, or dateTo is required. This prevents accidental exports of the complete literature index.

INSPIRE returns HTTP 400

The free-form query likely contains invalid native syntax, or a structured value produced an unsupported expression. Test a simpler topic query, then add filters one at a time. Invalid syntax is not retried.

The dataset is empty

Open the equivalent query on INSPIRE and confirm records currently match. Check author spelling, collaboration naming, subject capitalization, and the date range. A valid zero-match search completes successfully with no charged paper events.

Some metadata fields are empty

This normally means the official record does not contain that metadata. Use canonicalUrl to inspect the record and apiUrl to compare the underlying JSON.

A scheduled run failed after earlier success

The official service may be temporarily unavailable or rate-limiting requests. Keep bounded retries enabled and let the failed run signal your automation rather than accepting an incomplete snapshot.

Frequently asked questions

Does this Actor require an INSPIRE account?

No. It uses the public literature API without user credentials.

Can I search multiple criteria together?

Yes. The Actor joins your free query, author, collaboration, subject, and date clauses with and.

Can I export more than one API page?

Yes. Pagination continues automatically up to maxItems, with a maximum of 10,000 papers per run.

Does it return citing papers?

Every row includes citationsUrl. You can also use native syntax such as refersto:recid:451647 in query to export the citing literature itself.

Does it download PDFs?

No. It returns public links exposed by INSPIRE, which keeps runs lightweight and avoids unnecessary transfer.

Is the output real-time?

It is a snapshot of the official API response at retrievedAt, not a stream.

Related automation-lab Actors

• arXiv Paper Search & Export for arXiv-first topic, category, author, and date exports.
• PubMed Article Search Scraper for biomedical article metadata, abstracts, MeSH terms, and affiliations.

Choose this Actor when INSPIRE's high-energy-physics curation, citation graph, collaborations, subjects, and canonical literature IDs are the required source of truth.