DCAT / data.json Open-Data Catalog Harvester

A generic "point at any DCAT / Project-Open-Data data.json catalog" harvester. Under the U.S. Project Open Data mandate, every federal agency (and many state, city and international portals) publishes a machine-readable DCAT catalog at a …/data.json URL that lists every dataset it holds. They all share the identical envelope ({"@type": "dcat:Catalog", "dataset": [ … ]}), so one actor harvests the whole open-data estate — no per-portal scraper needed.

Give it a data.json URL (or several, or pick from the built-in agency registry), optionally filter by a keyword, and it turns each catalog entry into a clean, flat dataset row plus a lossless raw copy — streaming the file so even a 71 MB catalog is harvested in a few MB of RAM.

One actor, every DCAT catalog.

What it does

• Harvest a catalog — point dataUrl at any data.json and it walks the dataset[] array, emitting one row per dataset: title, description, identifier, accessLevel, issued / modified, publisher, contactPoint + contactEmail, landingPage, license, keyword[], theme[], the first downloadURL + format, the distributionCount, the full distribution[] array, and a lossless raw copy of the whole dataset object.
• Harvest many at once — pass a dataUrls list, an agencies list, or allAgencies to sweep several catalogs in one run. The maxRecords cap applies across all sources together.
• Keyword filter — set keyword to keep only datasets whose title, description, identifier, keyword[] or theme[] match (case-insensitive).
• Streams, never loads-all — the catalog is parsed incrementally, so huge feeds (e.g. NASA's 71 MB data.json) are safe and maxRecords stops early without downloading the rest.

Example inputs

Harvest the U.S. Department of Energy catalog (small, clean, ~480 datasets):

{ "dataUrl": "https://www.energy.gov/data.json", "maxRecords": 1000 }

Only climate-related datasets from NASA's (large) catalog:

{ "dataUrl": "https://data.nasa.gov/data.json", "keyword": "climate", "maxRecords": 500 }

Sweep several agencies from the built-in registry in one run:

{ "agencies": ["energy", "cdc", "nsf"], "maxRecords": 2000 }

Any DCAT catalog you like — point at your own:

{ "dataUrls": ["https://data.cdc.gov/data.json", "https://www.justice.gov/data.json"] }

Input reference

• dataUrl — a single DCAT / Project-Open-Data data.json URL.
• dataUrls — a list of data.json URLs to harvest together.
• agencies — pick from the built-in US-federal registry: energy, nasa, cdc, doj, nsf, opm, sba, treasury.
• allAgencies — harvest every agency in the built-in registry.
• keyword — optional case-insensitive keyword filter.
• maxRecords — total dataset cap across all sources (default 1000). Each returned dataset is one billable event.
• bearerToken / extraHeaders — optional, for gated catalogs; never required, never logged.

Tip: do not use https://catalog.data.gov/data.json — that aggregated endpoint 404s behind Cloudflare. Use the individual agency catalogs (like https://www.energy.gov/data.json), which are the actual Project-Open-Data source of record.

Output

One dataset row per catalog entry, with normalized columns plus:

• distribution — the full array of distributions (each a downloadURL / accessURL, format, mediaType, title, …), and downloadURL / format / distributionCount summarizing it.
• publisher (name) and publisherObject (the full nested organization), contactPoint + contactEmail.
• _sourceUrl / _catalogTitle — which catalog the row came from.
• raw — the complete, untouched DCAT dataset object.

A catalog with no matching datasets returns 0 rows, not an error. A wrong URL that returns a 404 / Cloudflare / HTML page fails fast with a clear message.

Pricing

Pay per dataset — one dataset event per row returned. A run that matches nothing costs nothing.

Notes

• Reads the DCAT-US / Project-Open-Data 1.1 data.json schema; scope is catalog metadata (what datasets exist and where to get them), not the underlying data files.
• Streams with an incremental JSON parser, so large catalogs never blow memory.
• Not affiliated with or endorsed by any agency or portal. Respect each provider's terms and any per-dataset licensing.