# Openaire Research Products Scraper (`real-estate-api/openaire-research-products-scraper`) Actor

OpenAIRE Research Products Scraper is a Python tool designed to query the OpenAIRE Graph API. It extracts metadata for publications, datasets, software, and other research outputs—including author details, access rights, and funding info—into structured formats for scientific analysis.

- **URL**: https://apify.com/real-estate-api/openaire-research-products-scraper.md
- **Developed by:** [real-estate-api](https://apify.com/real-estate-api) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.99 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## OpenAIRE Research Products Scraper: EU Research Outputs & Projects

<p align="center">
  <img src="https://imgur.com/2EpRA8Q.png" alt="OpenAIRE Research Products Scraper" width="800" />
</p>

Extract research publications, datasets, software records, and funded projects from the public OpenAIRE Graph API. Collect structured research metadata using keyword-based searches without an API key or login.

Export results as JSON, CSV, or Excel through the Apify run interface.

### Features

- Search research records using keywords.
- Retrieve research products or funded projects.
- Extract publication metadata, authors, subjects, funding information, and access rights.
- Convert HTML descriptions and project summaries into plain text.
- Skip duplicate records using OpenAIRE identifiers.
- Handle pagination, request timeouts, retries, and empty results.
- Save request errors and search information in structured dataset rows.
- Support scheduled runs for monitoring research outputs.

### What Data Does It Extract?

Each successful row represents a research product or project. The `productType` field identifies the record category, while project titles are stored in `mainTitle`.

| Field | Description |
|---|---|
| `recordId` | OpenAIRE record identifier used for deduplication. |
| `authors` | Author objects associated with a research product. |
| `openAccessColor` | Open-access color classification, when available. |
| `publiclyFunded` | Public funding indicator, when available. |
| `productType` | Record type, such as publication, dataset, software, or project. |
| `language` | Language code and label. |
| `countries` | Associated countries, when available. |
| `subjects` | Subject classification objects. |
| `mainTitle` | Research product title or project title. |
| `subTitle` | Subtitle, when available. |
| `descriptions` | Plain-text research product descriptions. |
| `publicationDate` | Publication date. |
| `publisher` | Publisher name. |
| `embargoEndDate` | End date of an access embargo. |
| `sources` | Source information. |
| `formats` | Available format information. |
| `contributors` | Contributor information. |
| `coverages` | Coverage information. |
| `bestAccessRight` | Access-right code, label, and scheme. |
| `container` | Container or publication venue information. |
| `documentationUrls` | Documentation references. |
| `codeRepositoryUrl` | Software repository reference. |
| `programmingLanguage` | Programming language information. |
| `contactPeople` | Associated contact people. |
| `contactGroups` | Associated contact groups. |
| `tools` | Related tools. |
| `projectCode` | Grant or project identifier. |
| `acronym` | Project acronym. |
| `websiteUrl` | Project website reference. |
| `startDate` | Project start date. |
| `endDate` | Project end date. |
| `callIdentifier` | Funding call identifier. |
| `keywords` | Project keywords. |
| `openAccessMandateForPublications` | Open-access mandate for publications. |
| `openAccessMandateForDataset` | Open-access mandate for datasets. |
| `fundings` | Funding objects. |
| `summaryText` | Plain-text project summary. |
| `sourceUrl` | API request reference associated with the row. |
| `scrapedAt` | UTC timestamp when the row was saved. |
| `error` | Error details, present on failed requests or empty matches when applicable. |
| `input` | Search and entity information associated with an error row. |

Optional fields may be absent when the source does not provide their values. Most returned values are preserved as supplied by OpenAIRE, except HTML descriptions and project summaries, which are converted to plain text.

### Use Cases

- **Literature and software discovery:** Build searchable spreadsheets of publications, datasets, and software matching a keyword.
- **EU-funded project research:** Collect project codes, acronyms, funding details, dates, and open-access mandates.
- **Open-access analysis:** Review access-right classifications alongside research titles.
- **Research monitoring:** Schedule recurring searches and compare record identifiers to identify new results.
- **Research data management:** Export structured records for literature reviews, lab notebooks, and funding research.

### Input Parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `search` | String | `python` | Keyword sent to the API search parameter. |
| `entity` | String | `researchProducts` | Record category: `researchProducts` or `projects`. |
| `maxItems` | Integer | `100` | Maximum number of successful records to save. Hard cap: 1,000. |
| `timeoutSecs` | Integer | `300` | Maximum run duration in seconds for starting new requests. |
| `requestTimeoutSecs` | Integer | `30` | Timeout for each individual request, in seconds. |

### Example Input

```json
{
  "search": "python",
  "entity": "researchProducts",
  "maxItems": 100,
  "timeoutSecs": 300,
  "requestTimeoutSecs": 30
}
```

To collect funded projects, change `entity` to `projects`. Adjust the keyword and limits to match your research task.

### Example Output

```json
{
  "recordId": "doi_dedup___::6ba3654ad017a5b7073fa7cd56",
  "authors": [
    {
      "fullName": "Onur Rauf Bingol",
      "rank": 1
    }
  ],
  "productType": "software",
  "language": {
    "code": "und",
    "label": "Undetermined"
  },
  "mainTitle": "NURBS-Python v3.6.0",
  "descriptions": [
    "The new version comes with performance improvements and documentation updates. It also adds bbox property for evaluation and storage of curve and surface bounding box."
  ],
  "publicationDate": "2017-06-20",
  "publisher": "Zenodo",
  "bestAccessRight": {
    "code": "c_abf2",
    "label": "OPEN",
    "scheme": "ACCESS_RIGHTS_SCHEME"
  },
  "codeRepositoryUrl": "CODE_REPOSITORY_URL",
  "scrapedAt": "2026-10-12T00:00:00Z",
  "sourceUrl": "API_REQUEST_URL"
}
```

The example uses placeholders for URL values. Actual output may contain additional fields depending on the record and the information returned by OpenAIRE.

### Pagination, Rate Limits, and Error Handling

The scraper queries the OpenAIRE Graph API for research products or projects. It starts at page 1 and follows subsequent pages until the reported result count is covered, a page returns no records, or `maxItems` is reached.

- **Batch size:** Requests up to 50 records per page, adjusted when fewer records are needed.
- **Sequential requests:** Requests run one at a time, with a 60-second wait before the next page.
- **Anonymous request pacing:** The scraper is configured for approximately one request per minute.
- **Retries:** HTTP 429 and 5xx responses, as well as request timeouts, are retried up to three additional times with waits of 2, 4, and 8 seconds.
- **Retry-After:** A longer `Retry-After` response value takes precedence over the standard retry wait.
- **Other failures:** A 4xx response or a successful HTTP response containing invalid JSON produces an error row.
- **Empty results:** An informational row is saved when a search returns no matches.
- **Duplicate handling:** Previously encountered record identifiers are skipped.

The maximum `maxItems` value is 1,000. Larger values are reduced to this limit.

Because pagination includes a 60-second interval, the default 300-second timeout may only allow the first page and a few subsequent pages. Increase `timeoutSecs` when collecting larger datasets. A request timeout applies to individual requests, while the run timeout limits how long the scraper continues initiating new requests.

OpenAIRE publishes different request allowances for anonymous and authenticated access. This scraper does not send an authentication token, so its request pacing is designed for anonymous use.

### Data Source and Licensing

The scraper retrieves records from the public OpenAIRE Graph API, using its research products and projects endpoints. No API key or login is required.

OpenAIRE data is published under a Creative Commons Attribution (CC BY) licence, according to the source information provided for this scraper. Verify the applicable terms and attribution requirements before redistributing or commercially reusing records.

The scraper respects its configured request pacing. Do not use proxies to circumvent source-imposed rate limits or access restrictions.

### Frequently Asked Questions

#### Does this scraper require an API key?

No. It uses public OpenAIRE Graph API endpoints without requiring a key or login.

#### Can I export the results to CSV, JSON, or Excel?

Yes. The Apify dataset can be downloaded in JSON, CSV, or Excel format.

#### How many records can one run collect?

The default is 100 records, and the maximum is 1,000 successful records per run. Larger searches may require a longer `timeoutSecs` value because requests are paced to avoid excessive traffic.

#### What happens if a search fails or returns no results?

The scraper records an error or informational row and completes the run instead of stopping unexpectedly.

#### Why is the record type field named `productType`?

The field stores the OpenAIRE record type. The name `productType` avoids using the generic field name `type` in the dataset schema. Funded project records are identified as projects.

#### Are all fields available for every record?

No. The fields returned depend on the type of record and the metadata supplied by OpenAIRE. Optional values may be missing.

### Changelog

**Version 1.0**

- Added support for research products and funded projects.
- Implemented pagination and record deduplication.
- Added structured error and informational rows.
- Added request timeout handling and retry logic.

### Trademark Notice

OpenAIRE and related names, logos, and trademarks belong to their respective owners. Their use in this documentation identifies the source of the data and does not imply affiliation, sponsorship, endorsement, or partnership.

### Disclaimer

This is an independent tool and is not affiliated with, endorsed by, sponsored by, or officially associated with OpenAIRE or its owners.

Data is collected from publicly available sources. Users are responsible for complying with applicable source terms, rate limits, attribution requirements, licensing conditions, and privacy laws.

### Is Web Scraping Legal?

Collecting publicly available information does not automatically make every subsequent use lawful. The legal requirements depend on the data collected, how it is used, and the terms that apply to the source.

- **Personal data:** Research records may contain names or other personal information. Applicable privacy laws, including the GDPR or CCPA where relevant, may impose requirements on collection, storage, and use.
- **Copyright:** Research descriptions and other content may be protected by copyright. Check the relevant licence before reproducing or redistributing content.
- **Terms and attribution:** Follow the source's usage terms, request limits, and attribution or licensing requirements.
- **Legal advice:** This section provides general information, not legal advice. Consult a qualified legal professional if you are unsure about your intended use.

# Actor input Schema

## `search` (type: `string`):

Keyword sent as the search parameter to the OpenAIRE Graph API. Example: python.

## `entity` (type: `string`):

researchProducts calls /graph/v1/researchProducts. projects calls /graph/v1/projects.

## `maxItems` (type: `integer`):

Maximum successful rows to save in one run. Values above 1000 are clamped to 1000.

## `timeoutSecs` (type: `integer`):

Stop scheduling new Graph API requests after this many seconds. Rows already saved are kept.

## `requestTimeoutSecs` (type: `integer`):

Per-request timeout in seconds for each call to the OpenAIRE Graph API.

## `proxyConfiguration` (type: `object`):

Optional proxy. Default is no proxy. The public Graph API does not need a proxy, and proxies are not used to get around stated limits.

## Actor input object example

```json
{
  "search": "python",
  "entity": "researchProducts",
  "maxItems": 100,
  "timeoutSecs": 300,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `datasetUrl` (type: `string`):

Default dataset for this run.

## `overview` (type: `string`):

Table of OpenAIRE research products or projects saved in this run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "search": "python",
    "entity": "researchProducts",
    "maxItems": 100,
    "timeoutSecs": 300,
    "requestTimeoutSecs": 30,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("real-estate-api/openaire-research-products-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "search": "python",
    "entity": "researchProducts",
    "maxItems": 100,
    "timeoutSecs": 300,
    "requestTimeoutSecs": 30,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("real-estate-api/openaire-research-products-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "search": "python",
  "entity": "researchProducts",
  "maxItems": 100,
  "timeoutSecs": 300,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call real-estate-api/openaire-research-products-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,real-estate-api/openaire-research-products-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/SI7HbpTur8BIy3cHx/builds/h1t8zcKh7WNlyFSxn/openapi.json