# DOE OSTI.gov Technical Reports Scraper (`crawlerbros/doe-osti-technical-reports-scraper`) Actor

Scrape the U.S. Department of Energy's OSTI.GOV public research repository - search millions of DOE-funded technical reports, journal articles, conference papers, patents, theses, and datasets by keyword, author, national lab, subject, or date range, or fetch a single record by OSTI ID.

- **URL**: https://apify.com/crawlerbros/doe-osti-technical-reports-scraper.md
- **Developed by:** [Crawler Bros](https://apify.com/crawlerbros) (community)
- **Categories:** Automation, Agents, Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## DOE OSTI.gov Technical Reports Scraper

Search the U.S. Department of Energy's public research repository, [OSTI.GOV](https://www.osti.gov) (Office of Scientific and Technical Information). Find DOE-funded technical reports, journal articles, conference papers, patents, theses/dissertations, program documents, scientific software, and research datasets — searchable by keyword, title, author, DOE national laboratory, sponsoring organization, subject, product type, and publication date range. Also supports direct lookup of a single record by its OSTI ID.

No login, API key, or cookies required — OSTI.GOV's `/api/v1/records` endpoint is a free, public REST API.

### What this actor does

- **Two modes:** `search` (query the full DOE research collection) and `byId` (fetch one record by OSTI ID)
- **Full-text and field search:** general keyword, title, author, sponsoring organization, research organization, subject/topic
- **DOE national laboratory quick filter:** one-click restriction to any of the 17 DOE national labs (Argonne, Los Alamos, Oak Ridge, NREL, Sandia, etc.)
- **Report type filter:** Technical Report, Journal Article, Conference paper, Patent, Book, Thesis/Dissertation, Program Document, Software, Dataset
- **Publication and entry date range filters**
- **Full-text availability filter**
- **Sort by publication date or entry date**
- **Empty fields are always omitted** — every field in the output is real, populated data

### Output per record

- `ostiId` — numeric OSTI identifier
- `title`
- `authors[]`, `primaryAuthor`
- `abstract` — plain-text description (HTML markup stripped)
- `researchOrganizations[]`, `primaryResearchOrganization` — performing lab/institution
- `sponsorOrganizations[]` — DOE program office / funding organization
- `contributingOrganizations[]` — secondary contributing organization(s), when disclosed
- `subjects[]` — subject categories and keywords
- `productType` — Technical Report / Journal Article / Conference / Patent / Book / Thesis-Dissertation / Program Document / Software / Dataset
- `articleType` — e.g. Accepted Manuscript, Published Article (journal articles only)
- `publicationDate`, `entryDate`
- `reportNumber`, `identifier`, `contractNumbers[]`, `otherIdentifiers[]`
- `doi`, `doiUrl`
- `journalName`, `journalVolume`, `journalIssue`, `journalIssn`, `journalSunsetDate`, `publisher` (journal articles)
- `language`, `countryOfPublication`
- `doeFunded` — whether the record is flagged as DOE-funded
- `sourceUrl` — canonical OSTI.GOV citation page
- `fulltextUrl` — direct PDF link for Technical Report / Journal Article records; for Dataset / Software records this instead resolves to the external data/code landing page (e.g. AmeriFlux, GitHub), when a full-text version is available
- `doePagesUrl` — DOE PAGES accepted-manuscript link, when available
- `dataExplorerUrl` — DOE Data Explorer link, when the record is a dataset
- `repositoryUrl`, `documentationUrl` — source-code repository / documentation link, when the record is Software or a linked Dataset
- `recordType: "report"`, `scrapedAt`

A `mode=byId` lookup for an invalid or unresolvable OSTI ID is still emitted as a typed record with `recordType: "error"` and an `error` message — it is never silently dropped.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `mode` | string | `search` | `search` or `byId` |
| `searchQuery` | string | – | General free-text query (mode=search). If every search filter is left empty, falls back to a demo query (`solar energy`) |
| `titleQuery` | string | – | Restrict to titles matching these terms |
| `authorQuery` | string | – | Author name |
| `sponsorOrg` | string | – | Sponsoring/funding organization |
| `researchOrg` | string | – | Performing research organization (lab, university, contractor) |
| `doeNationalLab` | select | – | Quick filter for one of the 17 DOE national laboratories |
| `subjectQuery` | string | – | Subject / topic keywords |
| `productType` | select | – | Technical Report / Journal Article / Conference / Patent / Book / Thesis-Dissertation / Program Document / Software / Dataset |
| `publicationDateStart` / `publicationDateEnd` | string | – | Publication date range, `YYYY-MM-DD` |
| `entryDateStart` / `entryDateEnd` | string | – | OSTI.GOV entry date range, `YYYY-MM-DD` |
| `language` | string | – | Publication language, e.g. `English`, `German` |
| `countryOfPublication` | string | – | Country of publication, e.g. `United States` |
| `requireFulltext` | boolean | `false` | Only records with a downloadable full-text PDF |
| `sortBy` | select | relevance | `publication_date` or `entry_date` |
| `sortOrder` | select | – | `asc` or `desc` |
| `ostiId` | string | – | OSTI ID to fetch (mode=byId), e.g. `3376627` (Software records use `code-<digits>`, e.g. `code-178644`) |
| `maxItems` | integer | `10` | Hard cap on emitted records (1–500) |

#### Example: search by keyword and organization

```json
{
  "mode": "search",
  "searchQuery": "battery storage",
  "doeNationalLab": "nrel",
  "productType": "Technical Report",
  "maxItems": 25
}
```

#### Example: publication date range for a subject

```json
{
  "mode": "search",
  "subjectQuery": "hydrogen fuel cells",
  "publicationDateStart": "2024-01-01",
  "publicationDateEnd": "2026-01-01",
  "maxItems": 50
}
```

#### Example: fetch one record by OSTI ID

```json
{
  "mode": "byId",
  "ostiId": "3376627"
}
```

### Use cases

- **Research intelligence** — track DOE-funded output by national laboratory, sponsor office, or subject
- **Energy policy analysis** — monitor technical reports and datasets on a topic (solar, hydrogen, nuclear, grid storage)
- **Bibliometrics** — collect journal-article and conference-paper metadata funded by DOE programs
- **Grant/contract tracking** — cross-reference DOE contract numbers to published outputs
- **Full-text mining** — bulk-collect direct PDF links for downstream text analysis

### FAQ

**Do I need an OSTI account or API key?**
No. OSTI.GOV's read/search API is fully public. (An account is only needed for E-Link, DOE's *submission* system for depositing new records — this actor only reads public data.)

**Why did my search return 0 results?**
Try broadening the query — combining a narrow `titleQuery` with a restrictive `productType` and a tight date range can over-constrain the search. `productType` and `requireFulltext` are applied after fetching, since OSTI.GOV's search API does not support them as native filters; very restrictive combinations may need a higher `maxItems` to see full results.

**What does `researchOrg` vs `doeNationalLab` do?**
`doeNationalLab` is a one-click shortcut for the 17 DOE national laboratories. `researchOrg` is free text and matches any performing organization (including universities and contractors) — set it directly if you need an organization not in the quick-filter list. If both are set, `researchOrg` takes priority.

**Is `fulltextUrl` always present?**
No — only records where OSTI.GOV hosts a full-text PDF include this field. Many journal-article records link out to the publisher/DOI instead; use `doiUrl` for those.

**What's the difference between `publicationDate` and `entryDate`?**
`publicationDate` is when the work was originally published. `entryDate` is when OSTI.GOV added or last updated the record in its system.

# Actor input Schema

## `mode` (type: `string`):

What to fetch.

## `searchQuery` (type: `string`):

General free-text query — searches the full record (title, abstract, authors, identifiers). Leave empty to search by another filter (author, organization, subject, date range, etc.) alone. If every search filter is left empty, the actor falls back to a demo query ("solar energy") so the daily test run still returns data.

## `titleQuery` (type: `string`):

Restrict results to records whose title matches these terms.

## `authorQuery` (type: `string`):

Search by author name, e.g. `Smith, John`.

## `sponsorOrg` (type: `string`):

Search the DOE sponsoring/funding organization field, e.g. `Office of Energy Efficiency and Renewable Energy`.

## `researchOrg` (type: `string`):

Search the performing research organization field (lab, university, contractor). Overrides `DOE national laboratory` if both are set.

## `doeNationalLab` (type: `string`):

Convenience shortcut for `Research organization` — restrict results to one of the 17 DOE national laboratories.

## `subjectQuery` (type: `string`):

Search DOE standardized subject categories and keywords, e.g. `photovoltaics`, `nuclear fuel cycle`, `battery storage`.

## `productType` (type: `string`):

Restrict results to one STI product type. Applied after fetching (OSTI.GOV does not support this as a server-side search parameter).

## `publicationDateStart` (type: `string`):

Earliest publication date, format `YYYY-MM-DD`.

## `publicationDateEnd` (type: `string`):

Latest publication date, format `YYYY-MM-DD`.

## `entryDateStart` (type: `string`):

Earliest date the record was added to OSTI.GOV, format `YYYY-MM-DD`.

## `entryDateEnd` (type: `string`):

Latest date the record was added to OSTI.GOV, format `YYYY-MM-DD`.

## `language` (type: `string`):

Filter by publication language, e.g. `English`, `German`, `Japanese`.

## `countryOfPublication` (type: `string`):

Filter by country of publication, e.g. `United States`, `Germany`.

## `requireFulltext` (type: `boolean`):

Only return records that have a downloadable full-text PDF on OSTI.GOV.

## `sortBy` (type: `string`):

Field to sort results by. Leave as default for relevance ranking.

## `sortOrder` (type: `string`):

Ascending or descending, used together with `Sort by`.

## `ostiId` (type: `string`):

Numeric OSTI identifier to fetch a single record, e.g. `3376627`. Software records use a `code-<digits>` identifier instead, e.g. `code-178644`.

## `maxItems` (type: `integer`):

Hard cap on emitted records (mode=search).

## Actor input object example

```json
{
  "mode": "search",
  "searchQuery": "solar energy",
  "doeNationalLab": "",
  "productType": "",
  "requireFulltext": false,
  "sortBy": "",
  "sortOrder": "",
  "maxItems": 10
}
```

# Actor output Schema

## `reports` (type: `string`):

Dataset containing all scraped OSTI.GOV records.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "mode": "search",
    "searchQuery": "solar energy",
    "doeNationalLab": "",
    "productType": "",
    "requireFulltext": false,
    "sortBy": "",
    "sortOrder": "",
    "maxItems": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("crawlerbros/doe-osti-technical-reports-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "mode": "search",
    "searchQuery": "solar energy",
    "doeNationalLab": "",
    "productType": "",
    "requireFulltext": False,
    "sortBy": "",
    "sortOrder": "",
    "maxItems": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("crawlerbros/doe-osti-technical-reports-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "mode": "search",
  "searchQuery": "solar energy",
  "doeNationalLab": "",
  "productType": "",
  "requireFulltext": false,
  "sortBy": "",
  "sortOrder": "",
  "maxItems": 10
}' |
apify call crawlerbros/doe-osti-technical-reports-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=crawlerbros/doe-osti-technical-reports-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/KlS0H85eXjvPhGb4F/builds/9IDfGbOA3QCpZxMRA/openapi.json