data.gov.au Dataset Scraper: Australian Open Data

Search the Australian government open data catalogue and save datasets or the files hanging off them. Results land in an Apify dataset you can export as JSON, CSV, or Excel.

A health search currently matches thousands of datasets. Each record carries the publisher, licence, contact, and the files already attached to it.

Use cases

• Catalogue research: pull Australian government datasets for a topic such as health, transport, or budget
• File inventory: switch to resources mode and list CSV, PDF, and other downloads with format and URL
• Licence review: keep license_id, license_title, and license_url next to the publisher
• Scheduled refresh: run the same query on a timer and diff metadata_modified
• Agency filter: narrow to a single publisher by organisation name

What data does the data.gov.au dataset scraper extract?

search writes one row per dataset. resources writes one row per file attached to those datasets.

Rows include the catalogue fields below, plus scrapedAt and sourceUrl. The public dataset title is stored as datasetTitle because title cannot be used as an Apify dataset field name. File format and file description are stored as resourceFormat and resourceDescription for the same reason.

Input

Parameter	Type	Default	Description
mode	string	search	search returns datasets. resources returns one row per file.
query	string	health	Search text to match datasets.
filterQuery	string	empty	Optional filter, such as an organisation name. A rejected filter is saved as an error row.
includeResources	boolean	true	In search mode, attach a slim file list on each dataset row.
maxItems	integer	100	Cap on dataset or resource rows. Hard maximum 1000.
timeoutSecs	integer	300	Stop the run after this many seconds.
requestTimeoutSecs	integer	30	Seconds to wait for each response before retrying.
proxyConfiguration	object	{"useApifyProxy": false}	Optional proxy. Leave it off unless you have a reason to send traffic elsewhere.

Example input

{
  "mode": "search",
  "query": "health",
  "filterQuery": "",
  "includeResources": true,
  "maxItems": 100,
  "timeoutSecs": 300,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": { "useApifyProxy": false }
}

Output

{
  "recordKind": "dataset",
  "archived": false,
  "author": "SA Health",
  "author_email": null,
  "contact_point": "dataandreportingservices@sa.gov.au",
  "creator_user_id": "c2fbbe4a-4ba0-4945-808b-67454605a4cf",
  "duplicate_score": 2,
  "geospatial_topic": [],
  "id": "a9372b1f-e5c0-4d03-9714-0dc707343f2f",
  "isopen": false,
  "language": "English",
  "license_id": "cc-by",
  "license_title": "Creative Commons Attribution 3.0 Australia",
  "license_url": "http://creativecommons.org/licenses/by/3.0/au/",
  "maintainer": "sharon-carthew-4072",
  "datasetTitle": "SA Health Mental Health data",
  "datasetName": "mental-health-data",
  "datasetPage": "https://data.gov.au/data/dataset/mental-health-data",
  "scrapedAt": "2026-10-12T00:00:00+00:00",
  "sourceUrl": "https://data.gov.au/"
}

Field	Type	Description
id	string	Dataset id, or resource id in resources mode. Used for dedupe.
author	string	Publisher or author from the catalogue.
contact_point	string	Contact published with the dataset.
license_id	string	Licence id on that dataset.
license_title	string	Licence name. Attribute the dataset the way that licence asks.
datasetTitle	string	Dataset title.
resources	array	Slim file list when includeResources is true.
resourceUrl	string	File URL on a resource row.
resourceFormat	string	File format label.
scrapedAt	string	UTC time the row was saved.
sourceUrl	string	Source catalogue URL for the row.
error	string	Present only when a row could not be saved.

Other catalogue fields are kept when the catalogue provides them: archived, author_email, creator_user_id, duplicate_score, geospatial_topic, isopen, language, license_url, maintainer, maintainer_email, metadata_created, metadata_modified, name, notes, num_resources, num_tags, organization, original_harvest_source, original_name, owner_org, private, promotion_level, remote_last_updated, spatial, and spatial_coverage.

Limits

maxItems stops the run at 1000 rows even if you type a higher number. Duplicate records are skipped. A search that matches nothing saves no rows. A rejected filter is saved as a single error row.

Contact fields sometimes contain a person's name or mailbox. Treat those as personal data and keep only what you have a reason to store.

Data source and licence

Source: data.gov.au, the Australian Government's open data catalogue.

The about page states: "Terms of Use Copyright With the exception of the Commonwealth Coat of Arms and where otherwise noted, material presented on Data.gov.au is provided under a Creative Commons Attribution 3.0 Australia licence." It also says: "With the exception of the Commonwealth Coat of Arms and where otherwise noted, this work is licensed under the CC BY 4.0 license."

The same page says: "When using content from this website that is licensed under a Creative Commons Licence, you are required to attribute the work in the manner specified in the licence". Each dataset row includes license_title and license_url. Follow the licence named on that row.

FAQ

What is the difference between search and resources? Search saves datasets. Resources saves one row per file attached to those datasets.

Why did I get an error row? A rejected filter or a row that could not be saved is stored as an error row instead of failing the whole run. A search that matches nothing saves zero rows and is not an error.

How many rows can one run save? Up to maxItems, with a hard cap of 1000. Raise maxItems to save more.

Can I export to CSV or Excel? Yes. The Apify dataset export covers JSON, CSV, and Excel.

Trademark Notice

data.gov.au Dataset and related names, logos, and trademarks belong to their respective owners. Their use in this documentation is solely to identify the source of the data and does not imply affiliation, sponsorship, endorsement, or partnership.

Disclaimer

Disclaimer: This Actor is an independent tool and is not affiliated with, endorsed by, sponsored by, or otherwise officially associated with data.gov.au Dataset or its owners. All data is collected from publicly available sources. Users are responsible for complying with the source's terms of use, rate limits, and attribution or licensing requirements.

Is web scraping legal?

This Actor only collects publicly available data from data.gov.au Dataset. Scraping public data is generally legal, but how you use it matters:

• Personal data: If the results include personal data (for example names, emails, or profile details), laws such as the GDPR in the EU and the CCPA in California may apply. Only collect and store personal data when you have a legitimate reason.
• Copyright: Some content (articles, images, descriptions) may be protected by copyright. Do not republish full copyrighted content without permission.
• Terms of use: Respect the source's terms, stated rate limits, and any attribution or non-commercial-use requirements listed in the Data source and licence section.
• Legal advice: This is general information, not legal advice. If you're unsure, consult a lawyer.

To learn more, read Apify's guide: Is web scraping legal?