🇺🇸 Data.gov Scraper

🚀 Collect open dataset metadata from Data.gov in minutes. Search by keyword, topic, or category. Export dataset titles, descriptions, download links, and agency info. No coding, no account required.

🕒 Last updated: 2026-04-23 · 📊 20+ fields per dataset · 🔍 3 search filters · 📂 Topic + category · 🚫 No auth required

The Data.gov Scraper collects open data catalog metadata from the U.S. government's data portal, returning 20+ fields per dataset: title, description, agency, topic, format, download URL, update frequency, and license. Data.gov hosts over 300,000 datasets from 100+ federal agencies.

The Actor supports keyword search with topic and category filters, plus direct URL scraping.

🎯 Target Audience	💡 Primary Use Cases
Data scientists, policy researchers, journalists, civic tech teams, academic researchers, government analysts	Open data discovery, policy research, data journalism, civic technology, federal data monitoring

---

📋 What the Data.gov Scraper does

Three search filters:

• 🔍 Keyword search. Free-text search across dataset titles and descriptions.
• 📂 Topic filter. Subject areas (health, education, climate, finance, etc.).
• 📂 Category filter. Detailed categorization within topics.
• 🔗 URL mode. Paste a direct Data.gov search URL.

Each dataset record includes title, description, agency, topic, categories, data format, download URLs, update frequency, license, and portal URL.

💡 Why it matters: browsing Data.gov for relevant datasets means scrolling through 300,000+ listings. This Actor exports structured catalog metadata at scale for your open data pipelines or research projects.

---

🎬 Full Demo

🚧 Coming soon: a 3-minute walkthrough showing how to go from sign-up to a downloaded dataset.

---

⚙️ Input

Input	Type	Default	Behavior
startUrl	string	""	Direct Data.gov search URL.
searchQuery	string	""	Keyword search.
topics	array	[]	Subject area filters.
topicCategories	array	[]	Detailed categories within topics.

Example: climate datasets.

{
    "searchQuery": "climate change",
    "topics": ["Climate"],
    "maxItems": 50
}

Example: health data from CDC.

{
    "searchQuery": "CDC",
    "topics": ["Health"],
    "maxItems": 100
}

⚠️ Good to Know: Data.gov is the U.S. government's official open data portal. Datasets link to various formats (CSV, JSON, XML, API) hosted by individual agencies.

---

📊 Output

Each dataset record contains 20+ fields. Download the catalog as CSV, Excel, JSON, or XML.

🧾 Schema

Field	Type	Example
📝 title	string	"Daily Climate Normals"
📄 description	string	"NOAA climate normals for U.S. stations..."
🏛️ agency	string	"NOAA"
📂 topic	string	"Climate"
📂 categories	array	["Weather", "Environment"]
📦 format	string	"CSV"
🔗 downloadUrl	string	"https://catalog.data.gov/dataset/..."
📅 lastUpdated	string	"2026-01-15"
🔄 updateFrequency	string	"Annual"
📜 license	string	"Public Domain"
🔗 portalUrl	string	"https://catalog.data.gov/dataset/..."
🕒 scrapedAt	ISO 8601	"2026-04-16T00:00:00.000Z"

📦 Sample records

---

✨ Why choose this Actor

	Capability
🇺🇸	300,000+ datasets. Full U.S. federal open data catalog.
🔍	3 search filters. Keyword, topic, category.
📦	Format and download links. Direct URLs to CSV, JSON, XML, API.
🏛️	Agency data. Federal agency per dataset.
📅	Update frequency. Daily, monthly, annual cadence.
⚡	Scalable. Quick lookups to full catalog sweeps.
🚫	No authentication. Public open data portal.

📊 Data.gov hosts 300,000+ datasets from 100+ federal agencies. Structured access powers every open data project, policy research, and civic technology workflow.

---

📈 How it compares to alternatives

Approach	Cost	Coverage	Refresh	Filters	Setup
⭐ Data.gov Scraper (this Actor)	$5 free credit, then pay-per-use	Full catalog	Live per run	keyword, topic, category	⚡ 2 min
Manual Data.gov browsing	Free	One at a time	Manual	UI only	🕒 Hours
CKAN API (direct)	Free	Full	Real-time	Many	⏳ Hours

Pick this Actor when you want U.S. federal open data catalog metadata on demand, with topic and category filters.

---

🚀 How to use

1. 📝 Sign up. Create a free account with $5 credit (takes 2 minutes).
2. 🌐 Open the Actor. Go to the Data.gov Scraper page on the Apify Store.
3. 🎯 Set input. Enter a keyword, pick topics and categories.
4. 🚀 Run it. Click Start.
5. 📥 Download. Grab results in the Dataset tab.

⏱️ Total time: 3-5 minutes. No coding required.

---

💼 Business use cases

📊 Data Science & Research Discover federal datasets for research Build open data catalogs Track new publications by agency Analyze data coverage by topic	🏛️ Policy & Civic Tech Monitor federal data releases Build transparency dashboards Track agency publishing rates Power civic apps with open data
📰 Data Journalism Find datasets for investigative reporting Track data freshness across agencies Build story pipelines from government data Monitor new dataset publications	🏢 Business Intelligence Enrich models with federal data Track economic indicators by agency Build market data pipelines Monitor regulatory data releases

---

---

🌟 Beyond business use cases

Data like this powers more than commercial workflows. The same structured records support research, education, civic projects, and personal initiatives.

🎓 Research and academia Empirical datasets for papers, thesis work, and coursework Longitudinal studies tracking changes across snapshots Reproducible research with cited, versioned data pulls Classroom exercises on data analysis and ethical scraping	🎨 Personal and creative Side projects, portfolio demos, and indie app launches Data visualizations, dashboards, and infographics Content research for bloggers, YouTubers, and podcasters Hobbyist collections and personal trackers
🤝 Non-profit and civic Transparency reporting and accountability projects Advocacy campaigns backed by public-interest data Community-run databases for local issues Investigative journalism on public records	🧪 Experimentation Prototype AI and machine-learning pipelines with real data Validate product-market hypotheses before engineering spend Train small domain-specific models on niche corpora Test dashboard concepts with live input

🤖 Ask an AI assistant about this scraper

Open a ready-to-send prompt about this ParseForge actor in the AI of your choice:

• 💬 ChatGPT
• 🧠 Claude
• 🔍 Perplexity
• 🅒 Copilot

💰 How much does it cost?

Apify gives you $5 in free monthly credits on the Apify Free plan, enough to test Data.gov Scraper and pull a real sample dataset. For ongoing usage:

• Starter plan ($49/month) — Recommended for individuals running Data.gov Scraper regularly. Includes higher concurrency and larger datasets.
• Scale plan ($499/month) — Recommended for teams running Data.gov Scraper at production scale.

Pay-Per-Event pricing means you only pay for what you actually use. Failed runs are never charged. See the Pricing tab on this Actor's page for exact event prices.

💡 Tips for using Data.gov Scraper

• Start with a small maxItems (3-10) to validate output format before running larger jobs.
• Use Apify Schedules to run Data.gov Scraper on a recurring basis and keep your dataset fresh.
• Export via Integrations: Apify connects to Google Sheets, Airbyte, Make, Zapier, and direct webhooks — pipe your data anywhere.
• Monitor with webhooks: trigger downstream workflows the moment a run finishes.
• Re-run failed items: if any individual records error out, re-run with their inputs only. Failed events are not charged.

⚖️ Is it legal to use Data.gov Scraper?

Yes. Data.gov Scraper only collects publicly available data. Web scraping public data has been confirmed as legal by US courts (see hiQ Labs v. LinkedIn) and is widely used for research, market analysis, and business intelligence.

However, you are responsible for:

• Respecting the source website's Terms of Service.
• Complying with GDPR, CCPA, and other applicable data-protection laws when personal data is involved.
• Not republishing copyrighted content without permission.

If you have specific compliance concerns, consult your legal team. See the Apify legal docs for more.

❓ Frequently Asked Questions

🔌 Automating Data.gov Scraper

• 🟢 Node.js. Install the apify-client NPM package.
• 🐍 Python. Use the apify-client PyPI package.
• 📚 See the Apify API documentation for full details.

The Apify Schedules feature lets you trigger this Actor on any cron interval. Weekly pulls catch new dataset publications.

🔌 Integrate with any app

• Make - Automate workflows
• Zapier - Connect 5,000+ apps
• Slack - Get notifications
• Airbyte - Data pipelines
• GitHub - Trigger from commits
• Google Drive - Export to Sheets

---

🔗 Recommended Actors

• 🇬🇧 Data.gov.uk Scraper - UK open data catalog
• 📊 USAspending Scraper - Federal spending data
• 📊 FRED Scraper - Economic data
• 📋 GSA eLibrary Scraper - Government contracts
• 📊 Indexmundi Scraper - Global indicators

💡 Pro Tip: browse the complete ParseForge collection for more government and open data scrapers.

---

🆘 Need Help? Open our contact form to request a new scraper, propose a custom data project, or report an issue.

---

⚠️ Disclaimer: this Actor is an independent tool and is not affiliated with, endorsed by, or sponsored by the U.S. government or Data.gov. All trademarks mentioned are the property of their respective owners. Only publicly available open data catalog metadata is collected.