US Government Datasets Scraper — Data.gov Catalog & Links avatar

US Government Datasets Scraper — Data.gov Catalog & Links

Pricing

from $4.00 / 1,000 data.gov dataset returneds

Go to Apify Store
US Government Datasets Scraper — Data.gov Catalog & Links

US Government Datasets Scraper — Data.gov Catalog & Links

Search Data.gov for US government datasets and export titles, publishers, tags, formats and download or API links for analysts. $5.00 per 1,000 datasets.

Pricing

from $4.00 / 1,000 data.gov dataset returneds

Rating

0.0

(0)

Developer

Muhammad Afzal

Muhammad Afzal

Maintained by Community

Actor stats

0

Bookmarked

1

Total users

1

Monthly active users

20 days ago

Last modified

Features

Limited permissions

Share

US Government Datasets Scraper searches the Data.gov catalog and returns structured metadata for government datasets: title, description, publisher, tags, file formats, download and API links, licence, access level, and dates. It is built for data analysts, researchers, journalists, and AI agents looking for official open data, and costs about $5.00 per 1,000 datasets on the FREE tier plus a $0.006251 start fee.

What data does the Data.gov scraper return?

It returns one record per matching Data.gov dataset. The record describes the dataset; it does not contain the dataset's rows.

FieldWhat it contains
titleDataset title as shown on Data.gov
descriptionCatalog description (may contain HTML)
datasetId, datasetNameStable identifier and catalog slug
datasetUrlLanding page published with the dataset, or its catalog.data.gov page
publisher, publisherIdPublishing organization and its Data.gov slug, for example noaa or nyc-ny
tagsKeywords attached to the dataset
formatsDistinct resource formats, such as TEXT/CSV or APPLICATION/JSON
resources[]Up to 50 download, API, or web links per dataset, each with name, format, url, description, lastModified
metadataCreated, metadataModifiedIssued date and last modified or harvested date
accessLevel, licenseAccess level (for example public) and licence when provided
sourceQuery, sourceApiUrl, scrapedAtYour query, the exact catalog request, and retrieval time

How do I search Data.gov datasets by keyword, agency, or tag?

Use searchQuery for words in titles, descriptions, tags, and publishers; add organization to restrict to one publisher slug and tags to require exact keywords.

{
"searchQuery": "air quality",
"organization": "epa",
"tags": ["air quality"],
"maxResults": 50,
"sortBy": "last_harvested_date",
"includeResources": true,
"maxResourcesPerDataset": 10
}
  • maxResults: 1–100 datasets per run (default 25).
  • pageSize: 1–100 records per catalog request (default 50).
  • sortBy: relevance (default), popularity, distance, or last_harvested_date.
  • includeResources: true (default) adds the resource links; false returns lighter metadata-only records.
  • maxResourcesPerDataset: 0–50 links per dataset (default 20).

Leave searchQuery empty to browse the whole catalog with your filters.

How do I get more than 100 government datasets?

Run again with the cursor. Each run stores nextAfter and hasMore in its OUTPUT record; pass nextAfter as after with the same filters to continue exactly where the previous run stopped, without paying for the first page twice.

Does it download the actual data files?

No. It returns each dataset's published links in resources[].url, which you can open, download, or pass to another tool. This keeps runs small and fast: the 25-dataset default run took about 5 seconds on 2026-10-26.

When should I use it, and when not?

Use it when you need:

  • a list of public datasets on a topic from US federal agencies and the state and city portals that publish to Data.gov;
  • the official download or API link for each dataset, with formats and update dates;
  • a scheduled catalog check to spot new or updated government datasets;
  • a compact, citable source list for an AI agent or RAG pipeline.

Do not use it when you need:

How do I call it from code?

Use the official Apify API clients with the same JSON input as in Console.

JavaScript (apify-client)

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('muhammadafzal/usa-data-gov-datasets-scraper').call({
searchQuery: 'public health',
maxResults: 25,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((d) => console.log(d.title, d.publisher, d.resources.map((r) => r.url)));

Python (apify-client 3.x)

import os
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("muhammadafzal/usa-data-gov-datasets-scraper").call(
run_input={"searchQuery": "public health", "maxResults": 25}
)
for dataset in client.dataset(run.default_dataset_id).iterate_items():
print(dataset["title"], dataset["publisher"], dataset["formats"])

Use it from an AI assistant

The Actor works from Claude, ChatGPT, Cursor, or any MCP client through Apify's MCP server. Add https://mcp.apify.com/?tools=muhammadafzal/usa-data-gov-datasets-scraper as a server, then ask, for example: "Find recent NOAA datasets about sea surface temperature on Data.gov and give me their CSV download links."

What does the output look like?

Records from the default climate run on 2026-10-26 (descriptions, tags, and resources shortened):

[
{
"datasetId": "https://data.cityofnewyork.us/api/views/c99a-c5ux",
"datasetName": "nyc-climate-budgeting-report-climate-alignment-assessment-and-capital-climate-investments",
"title": "NYC Climate Budgeting Report: Climate Alignment Assessment and Capital Climate Investments",
"description": "This dataset rates all projects in the capital budget with their climate alignment…",
"datasetUrl": "https://data.cityofnewyork.us/d/c99a-c5ux",
"publisher": "data.cityofnewyork.us",
"publisherId": "nyc-ny",
"tags": ["capital climate investments", "climate", "climate budgeting"],
"formats": ["APPLICATION/JSON", "APPLICATION/XML", "TEXT/CSV"],
"resources": [
{
"name": null,
"format": "application/json",
"url": "https://data.cityofnewyork.us/api/v3/views/c99a-c5ux/query.json?accessType=DOWNLOAD",
"description": null,
"lastModified": null
}
],
"metadataCreated": "2025-07-28",
"metadataModified": "2026-08-11",
"accessLevel": "public",
"license": null,
"sourceQuery": "climate",
"scrapedAt": "2026-10-26T07:49:28.566Z"
},
{
"datasetName": "state-of-the-climate-global-analysis",
"title": "State of the Climate - Global Analysis",
"publisherId": "noaa",
"formats": ["TEXT/HTML"],
"accessLevel": "public",
"license": "https://creativecommons.org/publicdomain/zero/1.0/"
}
]

How much does it cost to search government datasets?

You pay per dataset record delivered, plus one start fee. Platform usage is included in these prices.

EventFREEBRONZESILVERGOLD, PLATINUM, DIAMOND
Data.gov dataset returned$0.005001$0.004875$0.00475$0.004
Actor start (per started GB of allocated memory)$0.006251$0.006094$0.005938$0.005

The default memory is 256 MB, so a run normally pays one start event. Worked totals on the FREE tier:

  • Default run, 25 datasets: $0.006251 + 25 × $0.005001 = $0.131276.
  • Maximum run, 100 datasets: $0.006251 + 100 × $0.005001 = $0.506351 (GOLD: $0.405).

A dataset is charged once, after it is saved; duplicates are skipped. If a record only passes the dataset checks with its core fields (ID, name, title, link), it is saved without being charged and counted in OUTPUT.unchargedMinimalRecords. The start message and OUTPUT.estimatedPpeCostUsd use your run's live prices and include the start fee.

Limits and reliability

  • Up to 100 datasets per run; continue larger searches with after.
  • Each catalog request has a 30-second timeout and up to three attempts with backoff.
  • A valid search with no matches finishes successfully with zero records and no result charges.
  • If Data.gov rejects or rate-limits requests after retries, the run reports the error; if some datasets were already delivered, they are kept and the run ends as partial.

FAQ

Do I need a Data.gov API key?

No. There is no key field; the Actor handles access to Data.gov's Catalog API for you.

How do I download data from Data.gov with this Actor?

Run a search, then open or download the links in resources[].url. The Actor finds and describes the files; it does not fetch their contents.

Which organization slug should I use?

Use the publisher slug from Data.gov, the same value the Actor returns in publisherId, for example noaa, epa, or nyc-ny.

Does it cover state and city datasets?

Yes, when the state or city portal publishes to Data.gov. The sample run above includes New York City datasets alongside NOAA.

Can I monitor new government datasets?

Yes. Save an input with sortBy: "last_harvested_date" as a Task and schedule it; compare datasetId values between runs.

What formats can I export?

The dataset exports as JSON, CSV, Excel, XML, or RSS from Apify Console or the API.

The Actor reads Data.gov's public catalog metadata only. Each linked dataset has its own licence, access level, and terms; review them before reuse.

Last verified: 2026-10-27