DOE OSTI.gov Technical Reports Scraper

Search the U.S. Department of Energy's public research repository, OSTI.GOV (Office of Scientific and Technical Information). Find DOE-funded technical reports, journal articles, conference papers, patents, theses/dissertations, program documents, scientific software, and research datasets — searchable by keyword, title, author, DOE national laboratory, sponsoring organization, subject, product type, and publication date range. Also supports direct lookup of a single record by its OSTI ID.

No login, API key, or cookies required — OSTI.GOV's /api/v1/records endpoint is a free, public REST API.

What this actor does

• Two modes: search (query the full DOE research collection) and byId (fetch one record by OSTI ID)
• Full-text and field search: general keyword, title, author, sponsoring organization, research organization, subject/topic
• DOE national laboratory quick filter: one-click restriction to any of the 17 DOE national labs (Argonne, Los Alamos, Oak Ridge, NREL, Sandia, etc.)
• Report type filter: Technical Report, Journal Article, Conference paper, Patent, Book, Thesis/Dissertation, Program Document, Software, Dataset
• Publication and entry date range filters
• Full-text availability filter
• Sort by publication date or entry date
• Empty fields are always omitted — every field in the output is real, populated data

Output per record

• ostiId — numeric OSTI identifier
• title
• authors[], primaryAuthor
• abstract — plain-text description (HTML markup stripped)
• researchOrganizations[], primaryResearchOrganization — performing lab/institution
• sponsorOrganizations[] — DOE program office / funding organization
• contributingOrganizations[] — secondary contributing organization(s), when disclosed
• subjects[] — subject categories and keywords
• productType — Technical Report / Journal Article / Conference / Patent / Book / Thesis-Dissertation / Program Document / Software / Dataset
• articleType — e.g. Accepted Manuscript, Published Article (journal articles only)
• publicationDate, entryDate
• reportNumber, identifier, contractNumbers[], otherIdentifiers[]
• doi, doiUrl
• journalName, journalVolume, journalIssue, journalIssn, journalSunsetDate, publisher (journal articles)
• language, countryOfPublication
• doeFunded — whether the record is flagged as DOE-funded
• sourceUrl — canonical OSTI.GOV citation page
• fulltextUrl — direct PDF link for Technical Report / Journal Article records; for Dataset / Software records this instead resolves to the external data/code landing page (e.g. AmeriFlux, GitHub), when a full-text version is available
• doePagesUrl — DOE PAGES accepted-manuscript link, when available
• dataExplorerUrl — DOE Data Explorer link, when the record is a dataset
• repositoryUrl, documentationUrl — source-code repository / documentation link, when the record is Software or a linked Dataset
• recordType: "report", scrapedAt

A mode=byId lookup for an invalid or unresolvable OSTI ID is still emitted as a typed record with recordType: "error" and an error message — it is never silently dropped.

Input

Field	Type	Default	Description
mode	string	search	search or byId
searchQuery	string	–	General free-text query (mode=search). If every search filter is left empty, falls back to a demo query (solar energy)
titleQuery	string	–	Restrict to titles matching these terms
authorQuery	string	–	Author name
sponsorOrg	string	–	Sponsoring/funding organization
researchOrg	string	–	Performing research organization (lab, university, contractor)
doeNationalLab	select	–	Quick filter for one of the 17 DOE national laboratories
subjectQuery	string	–	Subject / topic keywords
productType	select	–	Technical Report / Journal Article / Conference / Patent / Book / Thesis-Dissertation / Program Document / Software / Dataset
publicationDateStart / publicationDateEnd	string	–	Publication date range, YYYY-MM-DD
entryDateStart / entryDateEnd	string	–	OSTI.GOV entry date range, YYYY-MM-DD
language	string	–	Publication language, e.g. English, German
countryOfPublication	string	–	Country of publication, e.g. United States
requireFulltext	boolean	false	Only records with a downloadable full-text PDF
sortBy	select	relevance	publication_date or entry_date
sortOrder	select	–	asc or desc
ostiId	string	–	OSTI ID to fetch (mode=byId), e.g. 3376627 (Software records use code-<digits>, e.g. code-178644)
maxItems	integer	10	Hard cap on emitted records (1–500)

Example: search by keyword and organization

{
  "mode": "search",
  "searchQuery": "battery storage",
  "doeNationalLab": "nrel",
  "productType": "Technical Report",
  "maxItems": 25
}

Example: publication date range for a subject

{
  "mode": "search",
  "subjectQuery": "hydrogen fuel cells",
  "publicationDateStart": "2024-01-01",
  "publicationDateEnd": "2026-01-01",
  "maxItems": 50
}

Example: fetch one record by OSTI ID

{
  "mode": "byId",
  "ostiId": "3376627"
}

Use cases

• Research intelligence — track DOE-funded output by national laboratory, sponsor office, or subject
• Energy policy analysis — monitor technical reports and datasets on a topic (solar, hydrogen, nuclear, grid storage)
• Bibliometrics — collect journal-article and conference-paper metadata funded by DOE programs
• Grant/contract tracking — cross-reference DOE contract numbers to published outputs
• Full-text mining — bulk-collect direct PDF links for downstream text analysis

FAQ

Do I need an OSTI account or API key? No. OSTI.GOV's read/search API is fully public. (An account is only needed for E-Link, DOE's submission system for depositing new records — this actor only reads public data.)

Why did my search return 0 results? Try broadening the query — combining a narrow titleQuery with a restrictive productType and a tight date range can over-constrain the search. productType and requireFulltext are applied after fetching, since OSTI.GOV's search API does not support them as native filters; very restrictive combinations may need a higher maxItems to see full results.

What does researchOrg vs doeNationalLab do? doeNationalLab is a one-click shortcut for the 17 DOE national laboratories. researchOrg is free text and matches any performing organization (including universities and contractors) — set it directly if you need an organization not in the quick-filter list. If both are set, researchOrg takes priority.

Is fulltextUrl always present? No — only records where OSTI.GOV hosts a full-text PDF include this field. Many journal-article records link out to the publisher/DOI instead; use doiUrl for those.

What's the difference between publicationDate and entryDate? publicationDate is when the work was originally published. entryDate is when OSTI.GOV added or last updated the record in its system.