# data.gov catalog search - datasets, resources, formats (API) (`retrainmap/data-gov-catalog`) Actor

Search the data.gov catalog (catalog.data.gov search API, no key) and get one row per dataset: id, title, organization, publisher, description, tags, formats, up to 10 resources with download URLs, license, modified date and landing page. Filter by query, organization, tags and file format.

- **URL**: https://apify.com/retrainmap/data-gov-catalog.md
- **Developed by:** [RetrainMap Team](https://apify.com/retrainmap) (community)
- **Categories:**
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 records

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## data.gov catalog search — datasets, resources, formats (API)

One row per dataset from the data.gov catalog, read through catalog.data.gov's public
search API. Filter by full-text query, organization, tags (keywords) and resource format;
get id, title, organization, publisher, description, tags, declared formats, up to 10
resources with download URLs, license, modified and last-harvested dates and the landing
page — normalised, one flat row per dataset, ready for CSV or JSON.

**The default input (`layoffs OR unemployment`, CSV resources, up to 200 rows) succeeds
within 5 minutes and returns rows** — Apify auto-tests it daily.

### Data source and status

| | |
|---|---|
| Publisher | U.S. General Services Administration (GSA), Technology Transformation Services — data.gov |
| API | https://catalog.data.gov/search — public, no key (OpenAPI description: https://catalog.data.gov/openapi.json; organizations list: https://catalog.data.gov/api/organizations) |
| What this Actor reads | Plain GET requests with `q`, `sort`, `keyword`, `org_slug`, `per_page=1000`; it follows the API's own `after` cursor, at most one request per second |
| Refresh cadence | data.gov re-harvests each source catalog on that source's schedule (most federal sources daily or weekly); every row carries the harvester's `last_harvested_date` and the publisher's own `metadata_modified`. The Actor reads live on every run; `fetched_at` is the fetch time |
| Copyright | The catalog metadata is a work of the United States Government and not subject to copyright in the U.S. (17 U.S.C. §105). Each **dataset** carries its own terms in the `license` column (public domain for most federal data; state, local and tribal publishers may use CC-BY or similar) — check it before reusing the data itself |
| Known caveats | The former CKAN endpoint (`catalog.data.gov/api/3/action/package_search`) answers HTTP 404 since the 2026 catalog rebuild — this Actor uses the successor search API; the API reports no total match count; there is no server-side format filter, so `formats` is applied after fetching (at most 20,000 results are scanned); a resource whose publisher declared neither `format` nor `mediaType` (typically an `accessURL` to a web page) has `format` null and never matches the format filter; `dataset_id` is the publisher's DCAT `identifier` (often a URL), `name` is the data.gov slug |

Honesty note: the rows are the API's own fields flattened — organization reduced to
name/slug/type, DCAT distributions to `{name, format, url}` and a list of declared
format labels (`text/csv` → `CSV`). Nothing is inferred from file names or URLs, nothing
is summarised or classified, and a run stops with an error rather than guess when the
API's shape changes.

Identification: the requests carry a product-token User-Agent and the operator's
contact address in the standard `From:` header (RFC 9110 §10.1.2).

### Input

| Field | Type | Meaning |
|---|---|---|
| `query` | string | Full-text query (`q`; OpenSearch query-string syntax: `AND`, `OR`, quoted phrases). Empty = all datasets |
| `organization` | string | Optional slug or exact name, e.g. `dol`, `census`, `noaa`, `iowa` (checked against the organizations list; unknown values fail with a hint) |
| `tags` | array | Optional keywords; a dataset must carry every listed keyword |
| `formats` | array | Optional, e.g. `CSV`, `JSON`, `XML`, `ZIP`, `GEOJSON`, `XLSX`, `PDF`, `HTML`, `API`; keep datasets declaring at least one resource in any listed format |
| `sort` | string | `relevance` (default), `popularity`, `last_harvested_date` |
| `max_records` | integer | Default 200, at most 5,000 |
| `request_interval_ms` | integer | Default 1000 (floor 250) |
| `contact_email` | string | Sent in the `From:` header |

Example — the Department of Labor's most recently harvested datasets about workforce
(no format filter: most DOL entries point at a web page with no declared file format):

```json
{ "query": "workforce", "organization": "dol", "sort": "last_harvested_date", "max_records": 25 }
```

### Output (dataset row)

`dataset_id`, `name` (data.gov slug), `title`, `organization`, `organization_slug`,
`organization_type`, `publisher`, `notes` (description, at most 2,000 characters), `tags`,
`themes`, `formats` (declared, upper-case labels), `resources` (first 10 `{name, format, url}`),
`resource_count`, `has_download`, `license`, `metadata_modified`, `issued`,
`last_harvested_date`, `popularity`, `landing_url`, `catalog_url`, `fetched_at`, `source`.

A run summary (filters, pages, results scanned, rows skipped by the format filter,
requests, whether the scan cap or the pay-per-event budget stopped the run) is stored as
`RUN_SUMMARY` in the run's key-value store.

### Pricing (pay per event)

| Event | Price |
|---|---|
| `run-start` — once per run | $0.10 |
| `record` — per row written | $0.005 |

The default pull (≤ 200 rows) costs at most $1.10; 1,000 datasets cost $5.10. Rows stop
when your run's maximum charge is reached; the run summary says so.

### Operator

Steelyard Ventures LLC (RetrainMap) — info@steelyardclinical.com. Not affiliated with GSA
or data.gov. The Actor writes only to its own dataset and key-value store; it stores no
credentials and sends nothing else.

# Actor input Schema

## `query` (type: `string`):

Full-text query passed as the API's q parameter (OpenSearch query-string syntax: AND / OR, double-quoted phrases). Empty = every dataset, in the chosen sort order.

## `organization` (type: `string`):

Optional. A data.gov organization slug such as dol, census, noaa, iowa, or its exact name. The Actor checks it against https://catalog.data.gov/api/organizations and fails with the list hint when it is unknown.

## `tags` (type: `array`):

Optional dataset keywords, e.g. unemployment, layoffs. Passed as the API's keyword filter; a dataset must carry every listed keyword.

## `formats` (type: `array`):

Optional. Keep only datasets that declare at least one resource in one of these formats, e.g. CSV, JSON, XML, ZIP, GEOJSON, XLSX, PDF, HTML, API (matched case-insensitively against each resource's declared format or media type, e.g. text/csv = CSV). The API has no format filter, so this is applied after fetching; the Actor scans at most 20,000 search results for matches.

## `sort` (type: `string`):

relevance (default), popularity, or last\_harvested\_date (newest harvest first).

## `max_records` (type: `integer`):

Stop after this many dataset rows have been written.

## `request_interval_ms` (type: `integer`):

Politeness delay towards catalog.data.gov. Each search request returns up to 1,000 datasets; the floor is 250 ms.

## `contact_email` (type: `string`):

Sent in the standard From: request header so the data owner can reach the operator (RFC 9110 s10.1.2).

## Actor input object example

```json
{
  "query": "layoffs OR unemployment",
  "formats": [
    "CSV"
  ],
  "sort": "relevance",
  "max_records": 200,
  "request_interval_ms": 1000,
  "contact_email": "info@steelyardclinical.com"
}
```

# Actor output Schema

## `results` (type: `string`):

Every matched data.gov dataset as one row: id, title, organization, publisher, description, tags, formats, resources, license, modified date, landing page.

## `results_csv` (type: `string`):

The same rows as a CSV file.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "layoffs OR unemployment",
    "formats": [
        "CSV"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("retrainmap/data-gov-catalog").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "query": "layoffs OR unemployment",
    "formats": ["CSV"],
}

# Run the Actor and wait for it to finish
run = client.actor("retrainmap/data-gov-catalog").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "layoffs OR unemployment",
  "formats": [
    "CSV"
  ]
}' |
apify call retrainmap/data-gov-catalog --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,retrainmap/data-gov-catalog"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/5PuhuIcqwYJXOmXTC/builds/EomsnE2vS1vHtJofW/openapi.json