OpenAIRE Research Products Scraper: EU Research Outputs & Projects

Extract research publications, datasets, software records, and funded projects from the public OpenAIRE Graph API. Collect structured research metadata using keyword-based searches without an API key or login.

Export results as JSON, CSV, or Excel through the Apify run interface.

Features

• Search research records using keywords.
• Retrieve research products or funded projects.
• Extract publication metadata, authors, subjects, funding information, and access rights.
• Convert HTML descriptions and project summaries into plain text.
• Skip duplicate records using OpenAIRE identifiers.
• Handle pagination, request timeouts, retries, and empty results.
• Save request errors and search information in structured dataset rows.
• Support scheduled runs for monitoring research outputs.

What Data Does It Extract?

Each successful row represents a research product or project. The productType field identifies the record category, while project titles are stored in mainTitle.

Field	Description
recordId	OpenAIRE record identifier used for deduplication.
authors	Author objects associated with a research product.
openAccessColor	Open-access color classification, when available.
publiclyFunded	Public funding indicator, when available.
productType	Record type, such as publication, dataset, software, or project.
language	Language code and label.
countries	Associated countries, when available.
subjects	Subject classification objects.
mainTitle	Research product title or project title.
subTitle	Subtitle, when available.
descriptions	Plain-text research product descriptions.
publicationDate	Publication date.
publisher	Publisher name.
embargoEndDate	End date of an access embargo.
sources	Source information.
formats	Available format information.
contributors	Contributor information.
coverages	Coverage information.
bestAccessRight	Access-right code, label, and scheme.
container	Container or publication venue information.
documentationUrls	Documentation references.
codeRepositoryUrl	Software repository reference.
programmingLanguage	Programming language information.
contactPeople	Associated contact people.
contactGroups	Associated contact groups.
tools	Related tools.
projectCode	Grant or project identifier.
acronym	Project acronym.
websiteUrl	Project website reference.
startDate	Project start date.
endDate	Project end date.
callIdentifier	Funding call identifier.
keywords	Project keywords.
openAccessMandateForPublications	Open-access mandate for publications.
openAccessMandateForDataset	Open-access mandate for datasets.
fundings	Funding objects.
summaryText	Plain-text project summary.
sourceUrl	API request reference associated with the row.
scrapedAt	UTC timestamp when the row was saved.
error	Error details, present on failed requests or empty matches when applicable.
input	Search and entity information associated with an error row.

Optional fields may be absent when the source does not provide their values. Most returned values are preserved as supplied by OpenAIRE, except HTML descriptions and project summaries, which are converted to plain text.

Use Cases

• Literature and software discovery: Build searchable spreadsheets of publications, datasets, and software matching a keyword.
• EU-funded project research: Collect project codes, acronyms, funding details, dates, and open-access mandates.
• Open-access analysis: Review access-right classifications alongside research titles.
• Research monitoring: Schedule recurring searches and compare record identifiers to identify new results.
• Research data management: Export structured records for literature reviews, lab notebooks, and funding research.

Input Parameters

Parameter	Type	Default	Description
search	String	python	Keyword sent to the API search parameter.
entity	String	researchProducts	Record category: researchProducts or projects.
maxItems	Integer	100	Maximum number of successful records to save. Hard cap: 1,000.
timeoutSecs	Integer	300	Maximum run duration in seconds for starting new requests.
requestTimeoutSecs	Integer	30	Timeout for each individual request, in seconds.

Example Input

{
  "search": "python",
  "entity": "researchProducts",
  "maxItems": 100,
  "timeoutSecs": 300,
  "requestTimeoutSecs": 30
}

To collect funded projects, change entity to projects. Adjust the keyword and limits to match your research task.

Example Output

{
  "recordId": "doi_dedup___::6ba3654ad017a5b7073fa7cd56",
  "authors": [
    {
      "fullName": "Onur Rauf Bingol",
      "rank": 1
    }
  ],
  "productType": "software",
  "language": {
    "code": "und",
    "label": "Undetermined"
  },
  "mainTitle": "NURBS-Python v3.6.0",
  "descriptions": [
    "The new version comes with performance improvements and documentation updates. It also adds bbox property for evaluation and storage of curve and surface bounding box."
  ],
  "publicationDate": "2017-06-20",
  "publisher": "Zenodo",
  "bestAccessRight": {
    "code": "c_abf2",
    "label": "OPEN",
    "scheme": "ACCESS_RIGHTS_SCHEME"
  },
  "codeRepositoryUrl": "CODE_REPOSITORY_URL",
  "scrapedAt": "2026-10-12T00:00:00Z",
  "sourceUrl": "API_REQUEST_URL"
}

The example uses placeholders for URL values. Actual output may contain additional fields depending on the record and the information returned by OpenAIRE.

Pagination, Rate Limits, and Error Handling

The scraper queries the OpenAIRE Graph API for research products or projects. It starts at page 1 and follows subsequent pages until the reported result count is covered, a page returns no records, or maxItems is reached.

• Batch size: Requests up to 50 records per page, adjusted when fewer records are needed.
• Sequential requests: Requests run one at a time, with a 60-second wait before the next page.
• Anonymous request pacing: The scraper is configured for approximately one request per minute.
• Retries: HTTP 429 and 5xx responses, as well as request timeouts, are retried up to three additional times with waits of 2, 4, and 8 seconds.
• Retry-After: A longer Retry-After response value takes precedence over the standard retry wait.
• Other failures: A 4xx response or a successful HTTP response containing invalid JSON produces an error row.
• Empty results: An informational row is saved when a search returns no matches.
• Duplicate handling: Previously encountered record identifiers are skipped.

The maximum maxItems value is 1,000. Larger values are reduced to this limit.

Because pagination includes a 60-second interval, the default 300-second timeout may only allow the first page and a few subsequent pages. Increase timeoutSecs when collecting larger datasets. A request timeout applies to individual requests, while the run timeout limits how long the scraper continues initiating new requests.

OpenAIRE publishes different request allowances for anonymous and authenticated access. This scraper does not send an authentication token, so its request pacing is designed for anonymous use.

Data Source and Licensing

The scraper retrieves records from the public OpenAIRE Graph API, using its research products and projects endpoints. No API key or login is required.

OpenAIRE data is published under a Creative Commons Attribution (CC BY) licence, according to the source information provided for this scraper. Verify the applicable terms and attribution requirements before redistributing or commercially reusing records.

The scraper respects its configured request pacing. Do not use proxies to circumvent source-imposed rate limits or access restrictions.

Frequently Asked Questions

Does this scraper require an API key?

No. It uses public OpenAIRE Graph API endpoints without requiring a key or login.

Can I export the results to CSV, JSON, or Excel?

Yes. The Apify dataset can be downloaded in JSON, CSV, or Excel format.

How many records can one run collect?

The default is 100 records, and the maximum is 1,000 successful records per run. Larger searches may require a longer timeoutSecs value because requests are paced to avoid excessive traffic.

What happens if a search fails or returns no results?

The scraper records an error or informational row and completes the run instead of stopping unexpectedly.

Why is the record type field named productType?

The field stores the OpenAIRE record type. The name productType avoids using the generic field name type in the dataset schema. Funded project records are identified as projects.

Are all fields available for every record?

No. The fields returned depend on the type of record and the metadata supplied by OpenAIRE. Optional values may be missing.

Changelog

Version 1.0

• Added support for research products and funded projects.
• Implemented pagination and record deduplication.
• Added structured error and informational rows.
• Added request timeout handling and retry logic.

Trademark Notice

OpenAIRE and related names, logos, and trademarks belong to their respective owners. Their use in this documentation identifies the source of the data and does not imply affiliation, sponsorship, endorsement, or partnership.

Disclaimer

This is an independent tool and is not affiliated with, endorsed by, sponsored by, or officially associated with OpenAIRE or its owners.

Data is collected from publicly available sources. Users are responsible for complying with applicable source terms, rate limits, attribution requirements, licensing conditions, and privacy laws.

Is Web Scraping Legal?

Collecting publicly available information does not automatically make every subsequent use lawful. The legal requirements depend on the data collected, how it is used, and the terms that apply to the source.

• Personal data: Research records may contain names or other personal information. Applicable privacy laws, including the GDPR or CCPA where relevant, may impose requirements on collection, storage, and use.
• Copyright: Research descriptions and other content may be protected by copyright. Check the relevant licence before reproducing or redistributing content.
• Terms and attribution: Follow the source's usage terms, request limits, and attribution or licensing requirements.
• Legal advice: This section provides general information, not legal advice. Consult a qualified legal professional if you are unsure about your intended use.