US Government Datasets Scraper — Data.gov Catalog & Links
Pricing
from $4.00 / 1,000 data.gov dataset returneds
US Government Datasets Scraper — Data.gov Catalog & Links
Search Data.gov for US government datasets and export titles, publishers, tags, formats and download or API links for analysts. $5.00 per 1,000 datasets.
Pricing
from $4.00 / 1,000 data.gov dataset returneds
Rating
0.0
(0)
Developer
Muhammad Afzal
Maintained by CommunityActor stats
0
Bookmarked
1
Total users
1
Monthly active users
20 days ago
Last modified
Categories
Features
Share
US Government Datasets Scraper searches the Data.gov catalog and returns structured metadata for government datasets: title, description, publisher, tags, file formats, download and API links, licence, access level, and dates. It is built for data analysts, researchers, journalists, and AI agents looking for official open data, and costs about $5.00 per 1,000 datasets on the FREE tier plus a $0.006251 start fee.
What data does the Data.gov scraper return?
It returns one record per matching Data.gov dataset. The record describes the dataset; it does not contain the dataset's rows.
| Field | What it contains |
|---|---|
title | Dataset title as shown on Data.gov |
description | Catalog description (may contain HTML) |
datasetId, datasetName | Stable identifier and catalog slug |
datasetUrl | Landing page published with the dataset, or its catalog.data.gov page |
publisher, publisherId | Publishing organization and its Data.gov slug, for example noaa or nyc-ny |
tags | Keywords attached to the dataset |
formats | Distinct resource formats, such as TEXT/CSV or APPLICATION/JSON |
resources[] | Up to 50 download, API, or web links per dataset, each with name, format, url, description, lastModified |
metadataCreated, metadataModified | Issued date and last modified or harvested date |
accessLevel, license | Access level (for example public) and licence when provided |
sourceQuery, sourceApiUrl, scrapedAt | Your query, the exact catalog request, and retrieval time |
How do I search Data.gov datasets by keyword, agency, or tag?
Use searchQuery for words in titles, descriptions, tags, and publishers; add organization to restrict to one publisher slug and tags to require exact keywords.
{"searchQuery": "air quality","organization": "epa","tags": ["air quality"],"maxResults": 50,"sortBy": "last_harvested_date","includeResources": true,"maxResourcesPerDataset": 10}
maxResults: 1–100 datasets per run (default 25).pageSize: 1–100 records per catalog request (default 50).sortBy:relevance(default),popularity,distance, orlast_harvested_date.includeResources:true(default) adds the resource links;falsereturns lighter metadata-only records.maxResourcesPerDataset: 0–50 links per dataset (default 20).
Leave searchQuery empty to browse the whole catalog with your filters.
How do I get more than 100 government datasets?
Run again with the cursor. Each run stores nextAfter and hasMore in its OUTPUT record; pass nextAfter as after with the same filters to continue exactly where the previous run stopped, without paying for the first page twice.
Does it download the actual data files?
No. It returns each dataset's published links in resources[].url, which you can open, download, or pass to another tool. This keeps runs small and fast: the 25-dataset default run took about 5 seconds on 2026-10-26.
When should I use it, and when not?
Use it when you need:
- a list of public datasets on a topic from US federal agencies and the state and city portals that publish to Data.gov;
- the official download or API link for each dataset, with formats and update dates;
- a scheduled catalog check to spot new or updated government datasets;
- a compact, citable source list for an AI agent or RAG pipeline.
Do not use it when you need:
- Canadian federal open data: use Canada Open Data Scraper;
- the rows inside a CSV or JSON file: download the
resources[].urllink instead; - research papers or citations: use Citation Count & DOI Citations Scraper.
How do I call it from code?
Use the official Apify API clients with the same JSON input as in Console.
JavaScript (apify-client)
import { ApifyClient } from 'apify-client';const client = new ApifyClient({ token: process.env.APIFY_TOKEN });const run = await client.actor('muhammadafzal/usa-data-gov-datasets-scraper').call({searchQuery: 'public health',maxResults: 25,});const { items } = await client.dataset(run.defaultDatasetId).listItems();items.forEach((d) => console.log(d.title, d.publisher, d.resources.map((r) => r.url)));
Python (apify-client 3.x)
import osfrom apify_client import ApifyClientclient = ApifyClient(os.environ["APIFY_TOKEN"])run = client.actor("muhammadafzal/usa-data-gov-datasets-scraper").call(run_input={"searchQuery": "public health", "maxResults": 25})for dataset in client.dataset(run.default_dataset_id).iterate_items():print(dataset["title"], dataset["publisher"], dataset["formats"])
Use it from an AI assistant
The Actor works from Claude, ChatGPT, Cursor, or any MCP client through Apify's MCP server. Add https://mcp.apify.com/?tools=muhammadafzal/usa-data-gov-datasets-scraper as a server, then ask, for example: "Find recent NOAA datasets about sea surface temperature on Data.gov and give me their CSV download links."
What does the output look like?
Records from the default climate run on 2026-10-26 (descriptions, tags, and resources shortened):
[{"datasetId": "https://data.cityofnewyork.us/api/views/c99a-c5ux","datasetName": "nyc-climate-budgeting-report-climate-alignment-assessment-and-capital-climate-investments","title": "NYC Climate Budgeting Report: Climate Alignment Assessment and Capital Climate Investments","description": "This dataset rates all projects in the capital budget with their climate alignment…","datasetUrl": "https://data.cityofnewyork.us/d/c99a-c5ux","publisher": "data.cityofnewyork.us","publisherId": "nyc-ny","tags": ["capital climate investments", "climate", "climate budgeting"],"formats": ["APPLICATION/JSON", "APPLICATION/XML", "TEXT/CSV"],"resources": [{"name": null,"format": "application/json","url": "https://data.cityofnewyork.us/api/v3/views/c99a-c5ux/query.json?accessType=DOWNLOAD","description": null,"lastModified": null}],"metadataCreated": "2025-07-28","metadataModified": "2026-08-11","accessLevel": "public","license": null,"sourceQuery": "climate","scrapedAt": "2026-10-26T07:49:28.566Z"},{"datasetName": "state-of-the-climate-global-analysis","title": "State of the Climate - Global Analysis","publisherId": "noaa","formats": ["TEXT/HTML"],"accessLevel": "public","license": "https://creativecommons.org/publicdomain/zero/1.0/"}]
How much does it cost to search government datasets?
You pay per dataset record delivered, plus one start fee. Platform usage is included in these prices.
| Event | FREE | BRONZE | SILVER | GOLD, PLATINUM, DIAMOND |
|---|---|---|---|---|
| Data.gov dataset returned | $0.005001 | $0.004875 | $0.00475 | $0.004 |
| Actor start (per started GB of allocated memory) | $0.006251 | $0.006094 | $0.005938 | $0.005 |
The default memory is 256 MB, so a run normally pays one start event. Worked totals on the FREE tier:
- Default run, 25 datasets: $0.006251 + 25 × $0.005001 = $0.131276.
- Maximum run, 100 datasets: $0.006251 + 100 × $0.005001 = $0.506351 (GOLD: $0.405).
A dataset is charged once, after it is saved; duplicates are skipped. If a record only passes the dataset checks with its core fields (ID, name, title, link), it is saved without being charged and counted in OUTPUT.unchargedMinimalRecords. The start message and OUTPUT.estimatedPpeCostUsd use your run's live prices and include the start fee.
Limits and reliability
- Up to 100 datasets per run; continue larger searches with
after. - Each catalog request has a 30-second timeout and up to three attempts with backoff.
- A valid search with no matches finishes successfully with zero records and no result charges.
- If Data.gov rejects or rate-limits requests after retries, the run reports the error; if some datasets were already delivered, they are kept and the run ends as partial.
FAQ
Do I need a Data.gov API key?
No. There is no key field; the Actor handles access to Data.gov's Catalog API for you.
How do I download data from Data.gov with this Actor?
Run a search, then open or download the links in resources[].url. The Actor finds and describes the files; it does not fetch their contents.
Which organization slug should I use?
Use the publisher slug from Data.gov, the same value the Actor returns in publisherId, for example noaa, epa, or nyc-ny.
Does it cover state and city datasets?
Yes, when the state or city portal publishes to Data.gov. The sample run above includes New York City datasets alongside NOAA.
Can I monitor new government datasets?
Yes. Save an input with sortBy: "last_harvested_date" as a Task and schedule it; compare datasetId values between runs.
What formats can I export?
The dataset exports as JSON, CSV, Excel, XML, or RSS from Apify Console or the API.
Related Actors
- Canada Open Data Scraper: Canada's Open Government portal.
- Airport Database Scraper: public-domain airport, runway, and navaid data.
- Open Access Books Scraper: DOAB open access book metadata.
- Citation Count & DOI Citations Scraper: citations and counts by DOI.
Legal and responsible use
The Actor reads Data.gov's public catalog metadata only. Each linked dataset has its own licence, access level, and terms; review them before reuse.
Last verified: 2026-10-27