OpenAIRE Research Search Scraper

This Actor searches publications and datasets through the official public OpenAIRE Graph API v3. It supports full-text or title-only search, publication-date and access filters, publication review status, official sort modes, bounded pagination, and fair global limits across selected result types.

Only normalized research products are written to the dataset. Empty searches and failures use the fixed OUTPUT summary, so diagnostic placeholders never appear beside results.

Example input

{
  "query": "climate change",
  "searchField": "all",
  "resultTypes": ["publications", "datasets"],
  "fromPublicationDate": "2024-01-01",
  "openAccessOnly": true,
  "peerReviewedOnly": false,
  "sortBy": "relevance",
  "sortDirection": "desc",
  "maxItems": 20,
  "pageSize": 10,
  "maxPages": 5,
  "maxConcurrency": 2,
  "maxRequestRetries": 2,
  "requestTimeoutSecs": 30
}

titleQuery remains as a backward-compatible alias that automatically selects title-only search. It cannot be combined with query.

The Actor maps inputs directly to documented OpenAIRE parameters:

• query → search or mainTitle
• resultTypes → separate type=publication / type=dataset requests
• fromPublicationDate, Open Access, and peer-review filters
• sortBy, direction, page, and pageSize

Pagination metadata and returned product types must match the request before records are accepted. Each response is also checked for HTTP status, final official host/path, JSON content type, maximum size, valid JSON, and documented payload shape. Temporary network, 429, and 5xx failures use bounded exponential backoff.

Output

Each row is a compact openaire-research-product record. It can include:

• OpenAIRE ID, product type, DOI or another persistent identifier
• title, subtitle, creators, compact description, and publication date
• access status, OA color, publisher, language, subjects, and countries
• citation count, funding/review flags, collection sources, and landing URL
• stable source request URL, source page/status, and scrape timestamp

Raw API objects, inventory metadata, internal actor metadata, signed URLs, credentials, and empty optional values are not stored. Records are deduplicated by OpenAIRE ID and buffered atomically. Multi-type results are interleaved page by page so publications cannot consume the global cap before datasets are considered.

Current API references

• OpenAIRE Graph API research products
• OpenAIRE paging
• OpenAIRE filtering

Local verification

npm ci
npm test
apify run --purge --input-file .actor/input.json

A clean no-result run leaves the dataset empty and sets status: NO_RESULTS. Invalid input fails before any API call.