# Data.gov Dataset Catalog Crawler (`jungle_synthesizer/datagov-dataset-crawler`) Actor Crawl 300K+ US government datasets from Data.gov. Extract titles, organizations, tags, formats, download URLs, API endpoints, temporal and spatial coverage, contacts, and resources. Filter by agency, format, category, and tags. - **URL**: https://apify.com/jungle\_synthesizer/datagov-dataset-crawler.md - **Developed by:** [BowTiedRaccoon](https://apify.com/jungle_synthesizer) (community) - **Categories:** Developer tools, Business, Other - **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing Pay per event This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Data.gov Dataset Catalog Crawler Extract dataset metadata from the [Data.gov](https://catalog.data.gov/) federal open data catalog. Covers 400K+ US government datasets — titles, descriptions, organizations, tags, formats, download URLs, API endpoints, temporal and spatial coverage, contact information, and individual resource details. ### Data.gov Crawler Features - Search across 400K+ federal datasets by keyword, organization, format, category, or tag - Filter by 20+ federal agencies including EPA, NASA, Census Bureau, HHS, DOE, and USDA - Filter by resource format: CSV, JSON, XML, API, Excel, PDF, Shapefile, GeoJSON, KML, RDF - Extract download URLs and API endpoints for every resource in a dataset - Includes temporal coverage (date ranges), spatial coverage (geographic bounds), and update frequency - Pulls contact names and emails for dataset maintainers - Toggle resource-level detail on or off — run lean for metadata, or include full file listings - Sort by relevance, recently modified, recently created, or alphabetically - Reads the CKAN v3 JSON API directly — no HTML parsing, no proxy required - Pay-per-event pricing at roughly $0.50 per 1,000 datasets ### Who Uses Data.gov Dataset Metadata? - **Data scientists and researchers** — discover federal datasets by topic, format, and time period without manually browsing catalog.data.gov - **Government contractors** — identify available datasets from specific agencies for proposal research and compliance reporting - **Journalists and civic technologists** — find public datasets on health, environment, transportation, or public safety for data-driven stories and apps - **Open data advocates** — monitor dataset freshness, publication rates, and coverage gaps across federal agencies - **B2B data teams** — build pipelines that pull structured government data feeds into internal analytics platforms ### How Data.gov Crawler Works 1. You configure search filters: keywords, organization, data format, category, tags, or any combination. 2. The crawler queries the Data.gov CKAN API with your filters and paginates through all matching results (1,000 per request, 200ms courtesy delay). 3. Each dataset record is transformed into a clean, flat output object with all metadata fields extracted and normalized. ### Input #### Browse all datasets (default) ```json { "maxItems": 100 } ```` #### Search by keyword ```json { "searchQuery": "air quality monitoring", "maxItems": 50 } ``` #### Filter by agency and format ```json { "organization": "environmental-protection-agency", "dataFormat": "CSV", "maxItems": 200 } ``` #### Filter by category and tags ```json { "category": "health", "tags": ["covid-19"], "maxItems": 100 } ``` #### Input Parameters | Field | Type | Default | Description | |-------|------|---------|-------------| | searchQuery | string | `""` | Full-text search across dataset titles, descriptions, and tags. Leave empty to browse all datasets. | | organization | string | `""` | Filter by publishing agency. Options include `environmental-protection-agency`, `department-of-commerce`, `national-aeronautics-and-space-administration`, `census-bureau`, and 16 others. | | dataFormat | string | `""` | Filter by resource format: CSV, JSON, XML, API, XLS, PDF, SHP, GeoJSON, KML, RDF, HTML. | | category | string | `""` | Filter by Data.gov topic: agriculture, business, climate, consumer, ecosystems, education, energy, finance, health, manufacturing, ocean, public-safety, science, weather. | | tags | string\[] | `[]` | Filter by dataset tags (e.g., `health`, `environment`). Datasets must match all specified tags. | | includeResources | boolean | `true` | Include individual resource details (file names, formats, URLs) within each dataset. Set to `false` for faster, leaner output. | | sortBy | string | `"score desc"` | Sort order: `score desc` (relevance), `metadata_modified desc` (recently modified), `metadata_created desc` (recently created), `name asc` (A-Z). | | maxItems | integer | `100` | Maximum dataset records to return. Set to `0` for unlimited. | ### Data.gov Crawler Output Fields #### Example Output ```json { "dataset_title": "Air Quality System (AQS) Data", "dataset_id": "c9310bc1-b224-4527-b3c2-0bd2bb24f455", "organization": "Environmental Protection Agency", "description": "The Air Quality System contains ambient air quality data collected by EPA, state, local, and tribal air pollution control agencies.", "tags": ["air-quality", "environment", "monitoring"], "categories": ["Climate", "Science & Research"], "update_frequency": "Annually", "temporal_coverage": "1980-01-01/2024-12-31", "spatial_coverage": "-180.0,18.0,-66.0,72.0", "formats": ["CSV", "API", "HTML"], "download_url": "https://aqs.epa.gov/aqsweb/airdata/download_files.html", "api_endpoint": "https://aqs.epa.gov/data/api", "license": "Public Domain", "modified_date": "2024-06-20", "created_date": "2014-10-09", "contact_name": "AQS Helpdesk", "contact_email": "aqs@epa.gov", "publisher": "US EPA Office of Air Quality", "resource_count": 3, "resources": [ "Data CSV | CSV | https://aqs.epa.gov/aqsweb/airdata/download_files.html", "API Endpoint | API | https://aqs.epa.gov/data/api", "Documentation | HTML | https://aqs.epa.gov/aqsweb/documents.html" ], "dataset_url": "https://catalog.data.gov/dataset/air-quality-system-aqs-data" } ``` #### Output Field Reference | Field | Type | Description | |-------|------|-------------| | dataset\_title | string | Title of the dataset | | dataset\_id | string | Unique CKAN dataset identifier | | organization | string | Publishing organization (federal agency name) | | description | string | Full description of the dataset | | tags | string\[] | Tags and keywords associated with the dataset | | categories | string\[] | Topic categories (climate, health, energy, etc.) | | update\_frequency | string | How often the dataset is updated (Daily, Monthly, Annually, etc.) | | temporal\_coverage | string | Time period covered (e.g., `2010-01-01/2023-12-31`) | | spatial\_coverage | string | Geographic area covered (coordinate bounds or description) | | formats | string\[] | Available resource formats (CSV, JSON, XML, API, etc.) | | download\_url | string | Direct download URL for the primary resource | | api\_endpoint | string | API endpoint URL if the dataset offers an API | | license | string | License type (Public Domain, Creative Commons, etc.) | | modified\_date | string | Date the metadata was last modified (YYYY-MM-DD) | | created\_date | string | Date the dataset was first published (YYYY-MM-DD) | | contact\_name | string | Dataset contact person or maintainer name | | contact\_email | string | Dataset contact email | | publisher | string | Publishing sub-agency or office | | resource\_count | number | Number of individual resources (files/APIs) in the dataset | | resources | string\[] | Resource details: `name | format | URL` (included when `includeResources` is true) | | dataset\_url | string | URL to the dataset page on catalog.data.gov | ### FAQ **How many datasets does Data.gov Crawler cover?** Data.gov Crawler queries the full Data.gov catalog — over 400,000 datasets published by federal agencies. If it is listed on catalog.data.gov, the crawler can find it. **Does this crawler need proxies?** No. Data.gov is a public government CKAN API with no authentication, no rate limits, and no bot detection. Proxies are disabled by default because they are genuinely unnecessary. **How fast does it run?** Roughly 1,000 datasets per 80 seconds. A 100-dataset run completes in under a minute on 256 MB memory. The full 400K+ catalog takes approximately 11 hours. **What is the difference between includeResources on and off?** With `includeResources` enabled (default), each dataset record lists every individual file and API resource — name, format, and URL. Disable it for faster runs when you only need dataset-level metadata. **Can I filter by multiple criteria at once?** Yes. Combine any of the filters — organization, format, category, tags, and search query. The CKAN API intersects all active filters, narrowing results to datasets that match every condition. **What data formats are available on Data.gov?** Common formats include CSV, JSON, XML, API, Excel (XLS/XLSX), PDF, Shapefile (SHP), GeoJSON, KML, RDF, and HTML. Filter by any of these to find datasets in the format you need. ### Need More Features? Need custom fields, additional filters, or a different government data source? [File an issue](https://console.apify.com/actors/issues) or get in touch. ### Why Use Data.gov Crawler? - **Full catalog access** — 400K+ federal datasets from 20+ agencies, searchable and filterable without navigating catalog.data.gov manually. - **Structured output** — Every record is normalized to the same flat schema with 21 fields, ready for downstream pipelines. - **Built on the official API** — Reads CKAN v3 JSON directly, so it does not break when someone redesigns a webpage. # Actor input Schema ## `sp_intended_usage` (type: `string`): Please describe how you plan to use the data extracted by this crawler. ## `sp_improvement_suggestions` (type: `string`): Provide any feedback or suggestions for improvements. ## `sp_contact` (type: `string`): Provide your email address so we can get in touch with you. ## `searchQuery` (type: `string`): Full-text search across dataset titles, descriptions, and tags. Leave empty to browse all datasets. ## `organization` (type: `string`): Filter by publishing organization (federal agency). ## `dataFormat` (type: `string`): Filter by resource format. Finds datasets containing at least one resource in the selected format. ## `tags` (type: `array`): Filter by dataset tags (e.g., 'health', 'environment'). Datasets must match all specified tags. ## `category` (type: `string`): Filter by Data.gov topic category. ## `includeResources` (type: `boolean`): When enabled, extracts individual resource (file/API) details within each dataset including download URLs, formats, and sizes. ## `sortBy` (type: `string`): Sort results by relevance, most recently modified, or metadata creation date. ## `maxItems` (type: `integer`): Maximum number of dataset records to return. Set to 0 for unlimited. ## `proxyConfiguration` (type: `object`): Select proxies. The Data.gov CKAN API is a government service and typically does not require proxies. ## Actor input object example ```json { "sp_intended_usage": "Describe your intended use...", "sp_improvement_suggestions": "Share your suggestions here...", "sp_contact": "Share your email here...", "tags": [], "includeResources": true, "sortBy": "score desc", "maxItems": 100, "proxyConfiguration": { "useApifyProxy": false } } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "sp_intended_usage": "Describe your intended use...", "sp_improvement_suggestions": "Share your suggestions here...", "sp_contact": "Share your email here...", "searchQuery": "", "organization": "", "dataFormat": "", "tags": [], "category": "", "includeResources": true, "sortBy": "score desc", "maxItems": 100, "proxyConfiguration": { "useApifyProxy": false } }; // Run the Actor and wait for it to finish const run = await client.actor("jungle_synthesizer/datagov-dataset-crawler").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "sp_intended_usage": "Describe your intended use...", "sp_improvement_suggestions": "Share your suggestions here...", "sp_contact": "Share your email here...", "searchQuery": "", "organization": "", "dataFormat": "", "tags": [], "category": "", "includeResources": True, "sortBy": "score desc", "maxItems": 100, "proxyConfiguration": { "useApifyProxy": False }, } # Run the Actor and wait for it to finish run = client.actor("jungle_synthesizer/datagov-dataset-crawler").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "sp_intended_usage": "Describe your intended use...", "sp_improvement_suggestions": "Share your suggestions here...", "sp_contact": "Share your email here...", "searchQuery": "", "organization": "", "dataFormat": "", "tags": [], "category": "", "includeResources": true, "sortBy": "score desc", "maxItems": 100, "proxyConfiguration": { "useApifyProxy": false } }' | apify call jungle_synthesizer/datagov-dataset-crawler --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=jungle_synthesizer/datagov-dataset-crawler", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Data.gov Dataset Catalog Crawler", "description": "Crawl 300K+ US government datasets from Data.gov. Extract titles, organizations, tags, formats, download URLs, API endpoints, temporal and spatial coverage, contacts, and resources. Filter by agency, format, category, and tags.", "version": "1.0", "x-build-id": "1AomDpW024f23zfeq" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/jungle_synthesizer~datagov-dataset-crawler/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-jungle_synthesizer-datagov-dataset-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/jungle_synthesizer~datagov-dataset-crawler/runs": { "post": { "operationId": "runs-sync-jungle_synthesizer-datagov-dataset-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/jungle_synthesizer~datagov-dataset-crawler/run-sync": { "post": { "operationId": "run-sync-jungle_synthesizer-datagov-dataset-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "sp_intended_usage", "sp_improvement_suggestions" ], "properties": { "sp_intended_usage": { "title": "What is the intended usage of this data?", "minLength": 1, "type": "string", "description": "Please describe how you plan to use the data extracted by this crawler." }, "sp_improvement_suggestions": { "title": "How can we improve this crawler for you?", "minLength": 1, "type": "string", "description": "Provide any feedback or suggestions for improvements." }, "sp_contact": { "title": "Contact Email", "minLength": 1, "type": "string", "description": "Provide your email address so we can get in touch with you." }, "searchQuery": { "title": "Search Query", "type": "string", "description": "Full-text search across dataset titles, descriptions, and tags. Leave empty to browse all datasets." }, "organization": { "title": "Organization", "enum": [ "", "department-of-health-and-human-services", "department-of-commerce", "department-of-agriculture", "environmental-protection-agency", "department-of-transportation", "department-of-energy", "department-of-defense", "department-of-the-interior", "department-of-labor", "national-aeronautics-and-space-administration", "department-of-education", "department-of-homeland-security", "department-of-justice", "department-of-housing-and-urban-development", "department-of-the-treasury", "department-of-veterans-affairs", "national-science-foundation", "small-business-administration", "general-services-administration", "census-bureau" ], "type": "string", "description": "Filter by publishing organization (federal agency)." }, "dataFormat": { "title": "Data Format", "enum": [ "", "CSV", "JSON", "XML", "API", "XLS", "PDF", "SHP", "GeoJSON", "KML", "RDF", "HTML" ], "type": "string", "description": "Filter by resource format. Finds datasets containing at least one resource in the selected format." }, "tags": { "title": "Tags", "type": "array", "description": "Filter by dataset tags (e.g., 'health', 'environment'). Datasets must match all specified tags.", "items": { "type": "string" } }, "category": { "title": "Category / Topic", "enum": [ "", "agriculture", "business", "climate", "consumer", "ecosystems", "education", "energy", "finance", "health", "local", "manufacturing", "maritime", "ocean", "public-safety", "science", "weather" ], "type": "string", "description": "Filter by Data.gov topic category." }, "includeResources": { "title": "Include Resource Details", "type": "boolean", "description": "When enabled, extracts individual resource (file/API) details within each dataset including download URLs, formats, and sizes.", "default": true }, "sortBy": { "title": "Sort By", "enum": [ "score desc", "metadata_modified desc", "metadata_created desc", "name asc" ], "type": "string", "description": "Sort results by relevance, most recently modified, or metadata creation date." }, "maxItems": { "title": "Max Items", "type": "integer", "description": "Maximum number of dataset records to return. Set to 0 for unlimited.", "default": 100 }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Select proxies. The Data.gov CKAN API is a government service and typically does not require proxies." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```