data.gov catalog search — datasets, resources, formats (API)

One row per dataset from the data.gov catalog, read through catalog.data.gov's public search API. Filter by full-text query, organization, tags (keywords) and resource format; get id, title, organization, publisher, description, tags, declared formats, up to 10 resources with download URLs, license, modified and last-harvested dates and the landing page — normalised, one flat row per dataset, ready for CSV or JSON.

The default input (layoffs OR unemployment, CSV resources, up to 200 rows) succeeds within 5 minutes and returns rows — Apify auto-tests it daily.

Data source and status

Publisher	U.S. General Services Administration (GSA), Technology Transformation Services — data.gov
API	https://catalog.data.gov/search — public, no key (OpenAPI description: https://catalog.data.gov/openapi.json; organizations list: https://catalog.data.gov/api/organizations)
What this Actor reads	Plain GET requests with q, sort, keyword, org_slug, per_page=1000; it follows the API's own after cursor, at most one request per second
Refresh cadence	data.gov re-harvests each source catalog on that source's schedule (most federal sources daily or weekly); every row carries the harvester's last_harvested_date and the publisher's own metadata_modified. The Actor reads live on every run; fetched_at is the fetch time
Copyright	The catalog metadata is a work of the United States Government and not subject to copyright in the U.S. (17 U.S.C. §105). Each dataset carries its own terms in the license column (public domain for most federal data; state, local and tribal publishers may use CC-BY or similar) — check it before reusing the data itself
Known caveats	The former CKAN endpoint (catalog.data.gov/api/3/action/package_search) answers HTTP 404 since the 2026 catalog rebuild — this Actor uses the successor search API; the API reports no total match count; there is no server-side format filter, so formats is applied after fetching (at most 20,000 results are scanned); a resource whose publisher declared neither format nor mediaType (typically an accessURL to a web page) has format null and never matches the format filter; dataset_id is the publisher's DCAT identifier (often a URL), name is the data.gov slug

Honesty note: the rows are the API's own fields flattened — organization reduced to name/slug/type, DCAT distributions to {name, format, url} and a list of declared format labels (text/csv → CSV). Nothing is inferred from file names or URLs, nothing is summarised or classified, and a run stops with an error rather than guess when the API's shape changes.

Identification: the requests carry a product-token User-Agent and the operator's contact address in the standard From: header (RFC 9110 §10.1.2).

Input

Field	Type	Meaning
query	string	Full-text query (q; OpenSearch query-string syntax: AND, OR, quoted phrases). Empty = all datasets
organization	string	Optional slug or exact name, e.g. dol, census, noaa, iowa (checked against the organizations list; unknown values fail with a hint)
tags	array	Optional keywords; a dataset must carry every listed keyword
formats	array	Optional, e.g. CSV, JSON, XML, ZIP, GEOJSON, XLSX, PDF, HTML, API; keep datasets declaring at least one resource in any listed format
sort	string	relevance (default), popularity, last_harvested_date
max_records	integer	Default 200, at most 5,000
request_interval_ms	integer	Default 1000 (floor 250)
contact_email	string	Sent in the From: header

Example — the Department of Labor's most recently harvested datasets about workforce (no format filter: most DOL entries point at a web page with no declared file format):

{ "query": "workforce", "organization": "dol", "sort": "last_harvested_date", "max_records": 25 }

Output (dataset row)

dataset_id, name (data.gov slug), title, organization, organization_slug, organization_type, publisher, notes (description, at most 2,000 characters), tags, themes, formats (declared, upper-case labels), resources (first 10 {name, format, url}), resource_count, has_download, license, metadata_modified, issued, last_harvested_date, popularity, landing_url, catalog_url, fetched_at, source.

A run summary (filters, pages, results scanned, rows skipped by the format filter, requests, whether the scan cap or the pay-per-event budget stopped the run) is stored as RUN_SUMMARY in the run's key-value store.

Pricing (pay per event)

Event	Price
run-start — once per run	$0.10
record — per row written	$0.005

The default pull (≤ 200 rows) costs at most $1.10; 1,000 datasets cost $5.10. Rows stop when your run's maximum charge is reached; the run summary says so.

Operator

Steelyard Ventures LLC (RetrainMap) — info@steelyardclinical.com. Not affiliated with GSA or data.gov. The Actor writes only to its own dataset and key-value store; it stores no credentials and sends nothing else.