Open Data Portal Harvester

🏛️ Open Data Portal Harvester is a powerful Apify Actor designed to discover, track, and collect datasets from multiple government open data portals including Data.gov (US) and Data.gov.uk (UK). This tool provides comprehensive Open Data Portal Harvester intelligence including dataset titles, download URLs, publishing organizations, licenses, and last-updated dates. Whether you're building a data catalog, monitoring government data releases, or conducting open data research, the Open Data Portal Harvester delivers actionable Open Data Portal Harvester results efficiently.

With multi-portal integration across Data.gov v4 and CKAN-based portals, keyword search, automatic resource format prioritization, and reliable Apify Dataset delivery, the Open Data Portal Harvester ensures comprehensive Open Data Portal Harvester coverage. It focuses on key Open Data Portal Harvester signals including dataset identifiers, download links, organizations, and licensing, making it an essential tool for open data research and government data intelligence.

---

📋 Table of Contents

• Features
• Data Sources
• How It Works
• Input
• Output
• Technical Stack
• Data Fields
• Use Cases
• Quick Start
• Configuration
• Performance
• Billing
• Important Notes
• Changelog
• Support
• License & Legal

---

🔥 Features

• Data.gov v4 Integration – Direct integration with the U.S. government's official open data API.
• CKAN Integration – Compatible with any CKAN-powered open data portal, including Data.gov.uk.
• Multi-Portal Intelligence – Aggregates Open Data Portal Harvester results from multiple sources in a single run.
• Keyword Search – Search datasets across portals by topic, agency, or subject.
• Automatic Pagination – Walks through result pages automatically until the limit or portal is exhausted.
• Smart Download URL Selection – Prioritizes CSV and JSON resource formats when available.
• Organization Tracking – Captures the publishing organization or agency for each dataset.
• License Capture – Records the dataset's license terms where available.
• Last-Updated Tracking – Captures the most recent modification date for each dataset.
• Resource Counting – Reports how many resources/files each dataset includes.
• Per-Portal Limiting – Configurable maximum datasets per portal to control run size.
• Residential Proxy Support – Optional Apify residential proxy for reliable access.
• Real-Time Dataset Push – Pushes results to Apify Dataset with metadata.
• Detailed Logging – Comprehensive logging of portal progress and dataset counts.
• Asyncio-Friendly – Non-blocking async/await architecture with httpx.
• Graceful Error Handling – Skips a failing portal without stopping the whole run.

---

📊 Data Sources

1. Data.gov (US)

• Authority: Official U.S. government open data portal
• API: Data.gov v4 Search API (api.gsa.gov/technology/datagov/v4)
• Coverage: Federal agency datasets across all sectors
• Data: Dataset metadata, DCAT distribution links, organization, license
• Access: Requires an API key (a DEMO_KEY works for light usage)

2. Data.gov.uk (UK)

• Authority: UK government open data portal
• API: CKAN Action API (/api/3/action/package_search)
• Coverage: UK government and public sector datasets
• Data: Dataset metadata, resource links, organization, license
• Access: No API key required

The Actor's CKAN support is portal-agnostic — any CKAN-based open data portal can be added by supplying its base URL.

---

⚙️ How It Works

The Open Data Portal Harvester accepts a search keyword and queries each configured portal using the appropriate API for its type — Data.gov v4 for datagov_v4 portals, and the CKAN Action API for ckan portals. For each dataset found, it extracts the title, dataset identifier, best available download URL, publishing organization, license, resource count, and last-updated date. Pagination continues automatically until the portal is exhausted or the configured per-portal limit is reached. Results from all portals are combined and pushed to the dataset in a single batch.

Key Processing Steps:

1. Input Parsing – Accept search keyword, per-portal limit, and API key
2. Proxy Setup – Optionally configure Apify residential proxy
3. Portal Loop – Iterate through each configured open data portal
4. Data.gov v4 Query – Paginate via after cursor for US datasets
5. CKAN Query – Paginate via start/rows for UK/CKAN datasets
6. Download URL Selection – Prefer CSV, then JSON, then any available link
7. Metadata Extraction – Capture organization, license, resource count, last updated
8. Timestamp Tagging – Record scrape time per dataset
9. Error Isolation – Log and skip any portal that fails without halting the run
10. Dataset Push – Push combined results in a single batch
11. Summary Logging – Report per-portal and total dataset counts

Key Benefits:

• Monitor new dataset releases across government open data portals
• Build a searchable catalog of public datasets
• Track dataset licensing and update frequency
• Compare data availability across US and UK sources
• Feed downstream research, journalism, or civic tech pipelines

---

📥 Input

The Actor accepts the following input parameters:

Field	Type	Default	Description
keyword	string	""	Search keyword for the Open Data Portal Harvester (e.g., "climate", "housing")
max_datasets_per_portal	integer	50	Maximum number of datasets to collect per portal (0 = no limit)
datagov_api_key	string	"DEMO_KEY"	API key for the Data.gov v4 Search API
useApifyProxy	boolean	true	Enable Apify Proxy
apifyProxyGroups	array	["RESIDENTIAL"]	Proxy group configuration

Example Input:

{
  "keyword": "climate change",
  "max_datasets_per_portal": 50,
  "datagov_api_key": "DEMO_KEY",
  "useApifyProxy": true
}

Search for Housing Datasets:

{
  "keyword": "housing",
  "max_datasets_per_portal": 100
}

---

📤 Output

The Actor pushes Open Data Portal Harvester records with the following structure:

Field	Type	Description
dataset_id	string	Unique identifier or slug for the dataset
title	string	Dataset title
download_url	string	Best available direct download link (CSV/JSON preferred)
source	string	Name of the originating portal
metadata.organization	string	Publishing organization or agency
metadata.license	string	Dataset license, where available
metadata.num_resources	integer	Number of resources/files attached to the dataset
metadata.last_updated	string	Date the dataset was last modified
metadata.notes	string	Truncated dataset description (up to 300 characters)
scraped_at	string	ISO 8601 timestamp of when the dataset was scraped

Example Open Data Portal Harvester Record:

{
  "dataset_id": "climate-risk-index-2025",
  "title": "National Climate Risk Index Dataset",
  "download_url": "https://catalog.data.gov/dataset/climate-risk-index-2025.csv",
  "source": "Data.gov (US)",
  "metadata": {
    "organization": "National Oceanic and Atmospheric Administration",
    "license": "Public Domain",
    "num_resources": 4,
    "last_updated": "2026-05-10",
    "notes": "County-level climate risk scores derived from historical weather and disaster data."
  },
  "scraped_at": "2026-07-25T12:00:00"
}

---

🧰 Technical Stack

• APIs: Data.gov v4 Search API, CKAN Action API (package_search)
• HTTP Client: httpx for async HTTP requests
• JSON Parsing: Standard json library
• Date Parsing: datetime for standardized timestamps
• Async: asyncio for non-blocking, paginated fetching
• Pagination: Cursor-based (after) for Data.gov v4, offset-based (start/rows) for CKAN
• Proxy: Apify Proxy with RESIDENTIAL configuration
• Logging: Apify Actor logging system
• Platform: Apify Actor serverless environment

---

📊 Data Fields Explained

Dataset Identification

• dataset_id: Unique identifier or slug for the dataset
• title: Dataset title
• source: Which portal the dataset came from

Access

• download_url: Best direct link to the dataset's data file

Publisher Info

• metadata.organization: Agency or organization that published the dataset
• metadata.license: License terms attached to the dataset

Content Summary

• metadata.num_resources: How many files/resources are attached
• metadata.last_updated: When the dataset was last modified
• metadata.notes: Short description excerpt

---

🎯 Use Cases

• Open Data Cataloging – Build a unified catalog of Open Data Portal Harvester results across portals
• Civic Tech Research – Power dashboards and apps built on public government data
• Journalism – Discover fresh datasets relevant to ongoing reporting
• Academic Research – Collect datasets for empirical studies
• Policy Analysis – Track newly published or updated government datasets
• Data Licensing Audits – Review license terms across many datasets at once
• Cross-Portal Comparison – Compare US vs. UK data availability on the same topic
• Dataset Monitoring – Re-run periodically to catch new or updated datasets

---

🚀 Quick Start

1. Prepare Input

Go to Apify Console and enter:

{
  "keyword": "renewable energy",
  "max_datasets_per_portal": 50
}

2. Run the Actor

Click Start. The Actor will:

• Query Data.gov (US) via the v4 Search API
• Query Data.gov.uk (UK) via the CKAN API
• Extract dataset metadata and download URLs
• Push combined results to the Dataset

3. Monitor Progress

Console shows:

Initializing Open Data Portal Harvester for Keyword: 'renewable energy'
[Data.gov v4] Fetching datasets from Data.gov (US)...
 -> Data.gov (US): 42 dataset(s) fetched.
[CKAN] Fetching datasets from Data.gov.uk (UK)...
 -> Data.gov.uk (UK): 37 dataset(s) fetched.
Done! Saved 79 datasets to Apify Dataset.

4. View & Download Results

• Results Tab: All Open Data Portal Harvester records
• Export: JSON, CSV, Excel, or HTML
• Filter: By source portal or organization
• API Access: Available via the Apify API

---

⚙️ Configuration

Broad topic search:

{
  "keyword": "population"
}

Narrow, high-volume search:

{
  "keyword": "air quality monitoring",
  "max_datasets_per_portal": 200
}

Custom Data.gov API key:

{
  "keyword": "transportation",
  "datagov_api_key": "your-api-key-here"
}

---

📈 Performance

Processing Speed

• Paginated fetching with a short delay between pages to respect rate limits
• ~50 datasets per page per portal
• Scales linearly with max_datasets_per_portal

Resource Usage

• Memory: Low-to-moderate, depending on result volume
• Network: Proportional to number of datasets and portals queried
• API calls: One paginated request cycle per portal

---

💰 Billing

Batch Billing

• Billing Model: Batch push (not per-dataset PPE)
• Typical Cost: Proportional to total datasets fetched across portals
• Efficiency: Pagination and per-portal limits keep runs predictable

---

⚠️ Important Notes

Legal & Compliance

• Fair Use: Respects each portal's public API and rate limits
• Attribution: Preserves the originating portal and organization for each dataset
• Accuracy: Data as recorded in the source portal's catalog
• Verification: Verify licensing terms directly with the source portal before reuse
• Legal: Not legal advice — review each dataset's license before redistribution

Data Quality

• Freshness: Reflects portal catalog state at time of run
• Completeness: Dependent on what each portal's API exposes
• Accuracy: Sourced directly from official government APIs
• Verification: Cross-check critical datasets against the live portal

Best Practices

• Use specific keywords to reduce noise in results
• Set a reasonable max_datasets_per_portal to control run size
• Check metadata.license before redistributing any dataset
• Re-run periodically to catch newly published datasets
• Supply your own Data.gov API key for higher rate limits than DEMO_KEY
• Add additional CKAN portals by extending the PORTALS list with their base URL

---

📦 Changelog

v1.0.0 (July 2026)

Initial Release:

• Data.gov v4 API integration
• CKAN Action API integration (Data.gov.uk and other CKAN portals)
• Keyword-based dataset search across portals
• Cursor-based pagination for Data.gov v4
• Offset-based pagination for CKAN portals
• Smart download URL selection (CSV/JSON preferred)
• Organization and license metadata extraction
• Resource count and last-updated tracking
• Per-portal dataset limiting
• Apify residential proxy support
• Real-time Dataset push
• Per-portal error isolation and logging
• Asyncio + httpx async architecture

---

🧑‍💻 Support & Feedback

• Issues: Submit via Apify console with the keyword used
• Documentation: Check the Actor details page
• Community: Apify forum discussions
• Feature Requests: Suggest additional open data portals to support
• Bug Reports: Include the keyword and error details

---

📄 License & Legal

Terms of Use:

• Use for legitimate open data research
• Respect each portal's Terms of Service and rate limits
• Review individual dataset licenses before redistribution
• Comply with applicable data protection and reuse laws
• Use data responsibly

Disclaimer: Open Data Portal Harvester is provided as-is for research purposes. Users are responsible for compliance with each source portal's ToS and licensing terms. This is not legal advice.

---

🎉 Get Started Today

Deploy now for Open Data Portal Harvester research!

Use for:

• 📊 Dataset Cataloging
• 🔍 Government Data Monitoring
• 💡 Civic Tech Projects
• 📈 Cross-Portal Comparison
• 🎯 Open Data Intelligence

Perfect for:

• Data Journalists
• Civic Technologists
• Researchers
• Policy Analysts
• Data Catalog Builders

---

Last Updated: July 2026 Version: 1.0.0 Status: Production Ready Platform: Apify Actor Architecture: Async/Await Data Sources: Data.gov (US), Data.gov.uk (UK), extensible CKAN portals

---

📚 Related Tools

• Public Company Job Postings Scraper
• Investor Presentation Scraper
• Salary Trends Scraper

---

⚖️ Open Data Portal Harvester Excellence

This Actor is optimized for Open Data Portal Harvester research with:

• ✅ Multi-portal Data.gov v4 and CKAN integration
• ✅ Smart download URL selection
• ✅ Organization and license tracking
• ✅ Automatic pagination
• ✅ Per-portal error isolation
• ✅ Real-time Dataset push
• ✅ Production-ready code