🦘 Australia Open Data (data.gov.au) Scraper

🚀 Export the Australian open data catalog in seconds. Pull 70,000+ datasets spanning federal, state, and territory agencies, including AEMO power market data, transport feeds, environmental records, and census tables. No API key, no manual CSV stitching, no portal click-through.

🕒 Last updated: 2026-05-22 · 📊 13 fields per record · 🦘 70,000+ datasets · 🏛️ Federal + 8 states/territories · ⚡ AEMO, ABS, BoM and more

The Australia Open Data Scraper taps the official data.gov.au catalog and returns 13 structured fields per dataset, including title, description, organization, license, resource files (CSV, GeoJSON, XLSX, ZIP, NetCDF), tags, and creation/modification timestamps. The portal aggregates publications from the Australian Bureau of Statistics, Australian Energy Market Operator, Bureau of Meteorology, Geoscience Australia, Department of Climate Change, and every state and territory government.

The catalog covers federal datasets, all eight states and territories, and major statutory authorities, with consistent licensing fields and downloadable resources. This Actor turns it into a clean dataset downloadable as CSV, Excel, JSON, or XML in under a minute. Filters and pagination run server-side, so you skip the parser engineering entirely.

🎯 Target Audience	💡 Primary Use Cases
ANZ researchers, fintech analysts, climate scientists, civic-tech builders, GIS engineers, journalists, NGOs	Open-data discovery, energy market analysis, climate research, regional dataset inventory, license auditing, ML training data

---

📋 What the Australia Open Data Scraper does

Two collection workflows in a single run:

• 🦘 Catalog walk. Paginate the full Australian dataset registry, optionally filtered by a search query.
• 🎯 Single dataset fetch. Retrieve one dataset by UUID or slug for targeted enrichment.
• 📁 Resource expansion. Every dataset includes its file resources with name, format, URL, language, size, and last-modified timestamp.
• 🏷️ Tag merge. All publication keywords consolidated into one clean list.
• 🏛️ Organization attribution. Publishing department, state agency, or statutory authority captured for every record.

Each record includes identifiers (UUID, slug), descriptive metadata (title, notes, tags), governance (organization, license, frequency), and a full list of downloadable resources.

💡 Why it matters: Australia's open data portal pools data from every level of government, but the portal UI is built for browsing one dataset at a time. This Actor turns it into a single downloadable spreadsheet so you can audit license terms, sort by recency, or build a research inventory in minutes.

---

🎬 Full Demo

🚧 Coming soon: a 3-minute walkthrough showing how to go from sign-up to a downloaded dataset.

---

⚙️ Input

Input	Type	Default	Behavior
maxItems	integer	10	Records to return. Free plan caps at 10, paid plan at 1,000,000.
mode	enum	"catalog"	catalog walks the registry, dataset fetches one record by ID.
datasetId	string	empty	UUID or slug of a single dataset. Required in dataset mode.
searchQuery	string	empty	Optional keyword filter applied in catalog mode.

Example: catalog walk of 100 datasets matching "energy".

{
    "maxItems": 100,
    "mode": "catalog",
    "searchQuery": "energy"
}

Example: fetch one dataset by slug.

{
    "mode": "dataset",
    "datasetId": "aemo-nem-dispatch-data"
}

⚠️ Good to Know: dataset frequency, license, and tag completeness vary by publishing agency. ABS and AEMO datasets are richly tagged; smaller council or local-authority datasets sometimes leave frequency blank. The Actor always returns the official portal URL so you can cross-check anything ambiguous.

---

📊 Output

Each dataset record contains up to 13 fields. Download the dataset as CSV, Excel, JSON, or XML.

🧾 Schema

Field	Type	Example
🆔 id	string	"d4b1f0fa-a5e8-4b9b-9d7a-1f3c5e6b1c2d"
🏷️ name	string	"aemo-nem-dispatch-data"
📰 title	string	"AEMO National Electricity Market Dispatch Data"
📝 notes	string	"5-minute dispatch prices and volumes..."
🏛️ organization	string	"Australian Energy Market Operator"
⚖️ license	string	"Creative Commons Attribution 4.0"
🏷️ tags	array	["electricity","NEM","dispatch","wholesale"]
📁 resourcesCount	number	5
📦 resources	array	[{name, format, url, language, size, lastModified}]
🕒 metadataCreated	ISO 8601	"2018-09-12T01:14:22.000Z"
🔁 metadataModified	ISO 8601	"2026-05-21T22:00:00.000Z"
📅 frequency	string | null	"hourly"
🔗 portalUrl	string	"https://data.gov.au/dataset/..."
🕒 scrapedAt	ISO 8601	"2026-05-22T00:00:00.000Z"

📦 Sample records

---

✨ Why choose this Actor

	Capability
🦘	All-of-government coverage. Federal, state, territory, and statutory authority datasets in one feed.
🎯	Two-mode workflow. Bulk catalog walk for inventory, single-dataset fetch for enrichment.
📁	Resource expansion. Each record carries its downloadable files (CSV, GeoJSON, ZIP, NetCDF) with size and last-modified date.
⚡	Energy-market ready. AEMO dispatch, demand, and forecast datasets surface in standard catalog mode.
🌦️	Climate-research ready. BoM and CSIRO observation datasets included with resource links.
🔁	Always fresh. Every run pulls the latest catalog state, no caching.
🚫	No authentication. Works on public open data. No API key needed.

📊 data.gov.au is the most comprehensive single source of free Australian government data. This Actor makes the entire registry queryable in minutes.

---

📈 How it compares to alternatives

Approach	Cost	Coverage	Refresh	Filters	Setup
⭐ Australia Open Data Scraper (this Actor)	$5 free credit, then pay-per-use	70,000+ Australian datasets	Live per run	Search keyword, single ID	⚡ 2 min
Hand-built CKAN client	Free	Up to you	Manual	Up to you	🐢 Days
Commercial data marketplaces	$$$	Curated subset	Variable	Vendor-defined	⏳ Hours
Manual portal browsing	Free	One at a time	Manual	UI search	🕒 Endless

Pick this Actor when you want a clean structured dataset of the entire Australian open data registry without writing a CKAN client.

---

🚀 How to use

1. 📝 Sign up. Create a free account with $5 credit (takes 2 minutes).
2. 🌐 Open the Actor. Go to the Australia Open Data Scraper page on the Apify Store.
3. 🎯 Set input. Pick catalog mode for a full walk or dataset mode with an ID, plus maxItems.
4. 🚀 Run it. Click Start and let the Actor collect your data.
5. 📥 Download. Grab your results in the Dataset tab as CSV, Excel, JSON, or XML.

⏱️ Total time from signup to downloaded dataset: 3-5 minutes. No coding required.

---

💼 Business use cases

⚡ Energy & Utilities Analytics Pull AEMO NEM and gas market datasets Track wholesale price publications Build forecasting inputs from official feeds Inventory renewable-energy publication coverage	🌦️ Climate & Environment Research Source BoM and CSIRO observation datasets Layer environmental files into GIS dashboards Build climate-impact studies with cited data Monitor air-quality and emissions publications
📰 Journalism & Civic Tech Audit which agencies publish what Track open-data publishing commitments Discover lesser-known transparency releases Surface licensing terms across jurisdictions	🏛️ Policy, Fintech & Research Aggregate ABS economic indicator publications Build cross-state analytics with citations Pull regulatory and reporting datasets Source baseline data for policy proposals

---

🔌 Automating Australia Open Data Scraper

Control the scraper programmatically for scheduled runs and pipeline integrations:

• 🟢 Node.js. Install the apify-client NPM package.
• 🐍 Python. Use the apify-client PyPI package.
• 📚 See the Apify API documentation for full details.

The Apify Schedules feature lets you trigger this Actor on any cron interval. Weekly or monthly refreshes keep downstream catalogs in sync automatically.

---

🌟 Beyond business use cases

Data like this powers more than commercial workflows. The same structured records support research, education, civic projects, and personal initiatives.

🎓 Research and academia Public-policy theses with reproducible data citations Climate-science publications using official observations Cross-jurisdiction open-government comparisons Course exercises on Australian data discovery	🎨 Personal and creative Dataviz portfolios sourced from ANZ data Indie civic dashboards and search interfaces Hobby projects on local transit or weather Content research for explainer blogs
🤝 Non-profit and civic NGO research on environmental and social files Civic-tech open-data discovery tools Investigative journalism around state releases Community map layers from official sources	🧪 Experimentation Train ML models on Australian government data Prototype agents that resolve dataset slugs Validate civic-tech product ideas with real metadata Build dataset search experiences over open feeds

---

🤖 Ask an AI assistant about this scraper

Open a ready-to-send prompt about this ParseForge actor in the AI of your choice:

• 💬 ChatGPT
• 🧠 Claude
• 🔍 Perplexity
• 🅒 Copilot

---

❓ Frequently Asked Questions

🧩 How does it work?

Configure your mode (catalog walk or single dataset) and optional keyword filter, click Start, and the Actor paginates the data.gov.au catalog server-side and emits one clean structured record per dataset.

🦘 How many datasets are in the catalog?

More than 70,000 datasets across federal, state, territory, and statutory agencies. The exact count changes as agencies publish, update, or retire records.

🔁 How often is the catalog refreshed?

Agencies publish on their own schedules (some hourly, some annually). Every run of this Actor fetches the latest catalog state in real time.

📁 Do I get the actual files or just the links?

You get the resource metadata for every file (name, format, URL, language, size, last-modified). Download the files themselves directly from the URLs in the resources field.

⚡ Can I get AEMO and BoM datasets through this Actor?

Yes. AEMO, Bureau of Meteorology, ABS, Geoscience Australia, CSIRO, and all state and territory agencies that publish to data.gov.au are in scope.

⏰ Can I schedule regular runs?

Yes. Use Apify Schedules to run this Actor on any cron interval (daily, weekly, monthly) and keep a downstream dataset inventory in sync.

⚖️ Is this data legal to use?

data.gov.au datasets are typically published under permissive Creative Commons licenses. Always review the per-dataset license field returned by this Actor before commercial reuse.

💼 Can I use this data commercially?

Most datasets allow commercial use under Creative Commons Attribution. A small number carry restrictions, which you can read in the license field for each record.

💳 Do I need a paid Apify plan to use this Actor?

No. The free Apify plan is enough for testing and small runs (10 records per run). A paid plan lifts the limit and gives you access to scheduling, higher concurrency, and larger datasets.

🔁 What happens if a run fails or gets interrupted?

Apify automatically retries transient errors. If a run still fails, you can inspect the log in the Runs tab, fix the input, and re-run. Partial datasets from failed runs are preserved so you never lose progress.

🆘 What if I need help?

Our support team is here to help. Contact us through the Apify platform or use the Tally form linked below.

---

🔌 Integrate with any app

Australia Open Data Scraper connects to any cloud service via Apify integrations:

• Make - Automate multi-step workflows
• Zapier - Connect with 5,000+ apps
• Slack - Get run notifications in your channels
• Airbyte - Pipe dataset metadata into your warehouse
• GitHub - Trigger runs from commits and releases
• Google Drive - Export datasets straight to Sheets

You can also use webhooks to trigger downstream actions when a run finishes. Push new datasets into your product backend, or alert your team in Slack.

---

🔗 Recommended Actors

• 🍁 Canada Open Data Scraper - Canadian federal open dataset catalog
• 🏢 ABR Australian Business Register Scraper - 7M+ Australian business records
• 📈 Indexmundi Scraper - Global demographic and economic indicators
• 🗺️ Nominatim OSM Scraper - Geocode addresses via OpenStreetMap
• ✈️ OurAirports Scraper - Global airport, heliport, and airfield dataset

💡 Pro Tip: browse the complete ParseForge collection for more reference-data scrapers.

---

🆘 Need Help? Open our contact form to request a new scraper, propose a custom data project, or report an issue.

---

⚠️ Disclaimer: this Actor is an independent tool and is not affiliated with, endorsed by, or sponsored by data.gov.au, the Australian Government, or any state or territory agency. All trademarks mentioned are the property of their respective owners. Only publicly available open government data is collected.