# data.gov.my Scraper - Malaysia Statistics & Open Data (`jungle_synthesizer/datagovmy-malaysia-statistics-scraper`) Actor Scrape data.gov.my, Malaysia's official open-data portal. Browse 300+ government datasets or pull time-series for specific ones with date filtering. Covers weekly fuel prices, CPI/PPI, labour stats, EPF, industrial production, and population data from DOSM, BNM, MOF and others. Clean JSON. No auth. - **URL**: https://apify.com/jungle\_synthesizer/datagovmy-malaysia-statistics-scraper.md - **Developed by:** [BowTiedRaccoon](https://apify.com/jungle_synthesizer) (community) - **Categories:** Business, Other - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing Pay per event This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## data.gov.my Scraper - Malaysia Statistics & Open Data Scrapes [data.gov.my](https://data.gov.my), Malaysia's official government open-data portal. Returns dataset metadata and time-series observations from DOSM, Bank Negara Malaysia, Ministry of Finance, Ministry of Health, and 40+ other agencies — weekly fuel prices, monthly CPI, labour stats, EPF demographics, industrial production, population by state, and 280+ more datasets, all in clean JSON. --- ### data.gov.my Scraper Features - **Two modes** — browse the full dataset catalogue (280+ entries with metadata) or pull time-series observations for specific datasets - **Date range filtering** — narrow any dataset to a specific window using `startDate` / `endDate` - **Publisher filtering** — limit catalogue browse to DOSM, BNM, MOF, MOH, or KKMM - **Covers the data people actually want** — weekly RON95/RON97/diesel fuel prices (the ones Malaysian media cites every Thursday), monthly CPI/PPI, LFS labour force, EPF statistics, IPI industrial production, and population tables - **No auth, no proxy** — the API is public and responds without credentials or residential IP tricks - **Multi-dataset runs** — supply a list of dataset IDs and the scraper pulls each in sequence --- ### Who Uses Malaysia Government Statistics? - **Malaysian media and research firms** — pull the canonical fuel-price or CPI numbers that NST, The Edge, and Malaysiakini reference in their reporting - **Corporate planning teams** — update FY models with the latest CPI/PPI/IPI without manually downloading CSV exports - **Fintech and consumer apps** — surface weekly RON95 cap prices that Petronas and industry trackers use - **ESG and sustainability reporters** — pull DOSM climate-station data and emissions-adjacent indicators - **Academia and think-tanks** — build long-horizon socioeconomic panels from DOSM's population, labour, and health datasets - **Developers and data engineers** — automate ingestion of government data into internal pipelines without managing the portal's Next.js interface --- ### How data.gov.my Scraper Works 1. **Set your mode.** `catalogue_browse` lists all 280+ available datasets with publisher, frequency, and tags. `dataset_pull` fetches actual observations for one or more dataset IDs you specify. 2. **Optionally filter.** In catalogue mode, narrow by publisher (DOSM, BNM, MOF, etc.). In dataset mode, pass a date range and the scraper filters to matching observation rows. 3. **The scraper fetches.** Catalogue mode parses the Next.js page for the embedded dataset collection. Dataset mode calls the REST API at `api.data.gov.my` with pagination, stopping when the page returns fewer records than the page size. 4. **You get structured JSON.** Each record carries `dataset_id`, `row_date`, `row_data` (all measure columns), and a direct `source_url` back to data.gov.my. --- ### Input **Dataset pull (fuel prices, specific year):** ```json { "mode": "dataset_pull", "datasetIds": ["fuelprice"], "startDate": "2025-01-01", "endDate": "2025-12-31", "maxItems": 100 } ```` **Catalogue browse (all DOSM datasets):** ```json { "mode": "catalogue_browse", "publisherFilter": "DOSM", "maxItems": 500 } ``` | Field | Type | Default | Description | |-------|------|---------|-------------| | `mode` | string | `dataset_pull` | `catalogue_browse` or `dataset_pull` | | `datasetIds` | string\[] | `["fuelprice", "cpi_headline"]` | Dataset IDs to pull (dataset\_pull mode only) | | `startDate` | string | — | Filter to observations on or after this date (YYYY-MM-DD) | | `endDate` | string | — | Filter to observations on or before this date (YYYY-MM-DD) | | `publisherFilter` | string | `any` | Limit catalogue to one publisher: `DOSM`, `BNM`, `MOF`, `MOH`, `KKMM`, or `any` | | `maxItems` | integer | 10 | Maximum number of records to return | *** ### data.gov.my Scraper Output Fields #### Dataset Pull Mode One record per observation row. The `row_data` field carries all measure columns for that dataset as a JSON string — for fuelprice that's `ron95`, `ron97`, `diesel`; for cpi that's the index components; and so on. ```json { "dataset_id": "fuelprice", "dataset_title": null, "publisher": null, "frequency": null, "last_updated": null, "description": null, "tags": null, "row_date": "2025-01-09", "row_data": "{\"ron95\":2.05,\"ron97\":3.47,\"diesel\":2.15,\"series_type\":\"level\"}", "unit": null, "source_url": "https://data.gov.my/data-catalogue/fuelprice", "scraped_at": "2025-01-15T08:30:00.000Z" } ``` | Field | Type | Description | |-------|------|-------------| | `dataset_id` | string | Canonical dataset ID (e.g. `fuelprice`, `cpi_headline`) | | `dataset_title` | string | Human-readable dataset title | | `publisher` | string | Publishing agency (populated in catalogue mode) | | `frequency` | string | Update frequency (populated in catalogue mode) | | `last_updated` | string | Last refresh ISO date (populated in catalogue mode) | | `description` | string | Catalogue description (populated in catalogue mode) | | `tags` | string | Categorical tags, pipe-separated (populated in catalogue mode) | | `row_date` | string | Observation date (YYYY-MM-DD) | | `row_data` | string | All measure columns as a JSON string | | `unit` | string | Measure unit | | `source_url` | string | Source page URL on data.gov.my | | `scraped_at` | string | Scrape timestamp ISO | #### Catalogue Browse Mode One record per dataset. Observation fields (`row_date`, `row_data`, `unit`) are null — this mode returns metadata only. ```json { "dataset_id": "fuelprice", "dataset_title": "Weekly Retail Fuel Prices", "publisher": "KPDN", "frequency": "weekly", "last_updated": "2025-01-09", "description": "Weekly retail fuel prices for RON95, RON97, and diesel in Malaysia.", "tags": "prices, fuel, energy", "row_date": null, "row_data": null, "unit": null, "source_url": "https://data.gov.my/data-catalogue/fuelprice", "scraped_at": "2025-01-15T08:30:00.000Z" } ``` *** ### 🔍 FAQ #### How do I find valid dataset IDs? Run the scraper in `catalogue_browse` mode first. It returns all 280+ datasets with their IDs, publishers, and descriptions. The IDs you see there — `fuelprice`, `cpi_headline`, `lfs_month`, `epf_demographics` — are what you pass to `datasetIds` in dataset pull mode. #### How much does data.gov.my Scraper cost to run? At the default PPE rate, a typical dataset pull (100–500 rows) costs fractions of a cent. Full fuel price history (900+ weekly records) runs in seconds. The API is free and fast, so costs are dominated by Apify's minimum start fee rather than record count. #### Does data.gov.my Scraper need proxies? No. The data.gov.my API is public and doesn't block by geography or IP. A standard browser User-Agent header is sufficient. No residential proxies, no CAPTCHA. #### What datasets does data.gov.my actually have? Quite a few. Weekly fuel prices, monthly CPI/PPI/IPI, quarterly GDP components, annual population by state and district, monthly labour force survey, EPF and SOCSO statistics, electricity tariff data, climate station readings, road accidents, crime by state, health indicators, education enrollment. Run catalogue browse to see the full list — it expands as agencies publish new datasets. #### Can I pull multiple datasets in one run? Yes. Pass a list to `datasetIds`: `["fuelprice", "cpi_headline", "lfs_month"]`. The scraper fetches each in sequence and combines the output. The `dataset_id` field on each record tells you which dataset it came from. *** ### Need More Features? Need additional metadata fields, webhook delivery for weekly fuel updates, or a specific output format? [File a feature request](https://console.apify.com/actors/GYmVYdS7A5rLaiOE4/issues) or contact the team. ### Why Use data.gov.my Scraper? - **First and only** — no competitors on Apify target the data.gov.my v2 API, which launched in 2023 and keeps adding datasets. - **Covers the numbers that get cited** — fuel prices, CPI, labour stats, and EPF data are the canonical government figures that Malaysian media, corporates, and academics reference. Programmatic access to the same source, without the CSV downloads. - **No friction** — public API, no auth, no proxy, no CAPTCHA. You configure it once and it runs. # Actor input Schema ## `sp_intended_usage` (type: `string`): Please describe how you plan to use the data extracted by this crawler. ## `sp_improvement_suggestions` (type: `string`): Provide any feedback or suggestions for improvements. ## `sp_contact` (type: `string`): Provide your email address so we can get in touch with you. ## `mode` (type: `string`): No description ## `datasetIds` (type: `array`): Canonical IDs from the data.gov.my catalogue (e.g. 'fuelprice', 'cpi\_headline'). Leave empty for catalogue browse mode. ## `startDate` (type: `string`): Filter observations on or after this date. ## `endDate` (type: `string`): Filter observations on or before this date. ## `publisherFilter` (type: `string`): No description ## `maxItems` (type: `integer`): Maximum number of records to return. ## Actor input object example ```json { "sp_intended_usage": "Describe your intended use...", "sp_improvement_suggestions": "Share your suggestions here...", "sp_contact": "Share your email here...", "mode": "dataset_pull", "datasetIds": [ "fuelprice", "cpi_headline" ], "publisherFilter": "any", "maxItems": 10 } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "sp_intended_usage": "Describe your intended use...", "sp_improvement_suggestions": "Share your suggestions here...", "sp_contact": "Share your email here...", "mode": "dataset_pull", "datasetIds": [ "fuelprice", "cpi_headline" ], "startDate": "", "endDate": "", "publisherFilter": "any", "maxItems": 10 }; // Run the Actor and wait for it to finish const run = await client.actor("jungle_synthesizer/datagovmy-malaysia-statistics-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "sp_intended_usage": "Describe your intended use...", "sp_improvement_suggestions": "Share your suggestions here...", "sp_contact": "Share your email here...", "mode": "dataset_pull", "datasetIds": [ "fuelprice", "cpi_headline", ], "startDate": "", "endDate": "", "publisherFilter": "any", "maxItems": 10, } # Run the Actor and wait for it to finish run = client.actor("jungle_synthesizer/datagovmy-malaysia-statistics-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "sp_intended_usage": "Describe your intended use...", "sp_improvement_suggestions": "Share your suggestions here...", "sp_contact": "Share your email here...", "mode": "dataset_pull", "datasetIds": [ "fuelprice", "cpi_headline" ], "startDate": "", "endDate": "", "publisherFilter": "any", "maxItems": 10 }' | apify call jungle_synthesizer/datagovmy-malaysia-statistics-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=jungle_synthesizer/datagovmy-malaysia-statistics-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "data.gov.my Scraper - Malaysia Statistics & Open Data", "description": "Scrape data.gov.my, Malaysia's official open-data portal. Browse 300+ government datasets or pull time-series for specific ones with date filtering. Covers weekly fuel prices, CPI/PPI, labour stats, EPF, industrial production, and population data from DOSM, BNM, MOF and others. Clean JSON. No auth.", "version": "0.1", "x-build-id": "DHS1e7gehIygtrFGg" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/jungle_synthesizer~datagovmy-malaysia-statistics-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-jungle_synthesizer-datagovmy-malaysia-statistics-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/jungle_synthesizer~datagovmy-malaysia-statistics-scraper/runs": { "post": { "operationId": "runs-sync-jungle_synthesizer-datagovmy-malaysia-statistics-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/jungle_synthesizer~datagovmy-malaysia-statistics-scraper/run-sync": { "post": { "operationId": "run-sync-jungle_synthesizer-datagovmy-malaysia-statistics-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "sp_intended_usage", "sp_improvement_suggestions" ], "properties": { "sp_intended_usage": { "title": "What is the intended usage of this data?", "minLength": 1, "type": "string", "description": "Please describe how you plan to use the data extracted by this crawler." }, "sp_improvement_suggestions": { "title": "How can we improve this crawler for you?", "minLength": 1, "type": "string", "description": "Provide any feedback or suggestions for improvements." }, "sp_contact": { "title": "Contact Email", "minLength": 1, "type": "string", "description": "Provide your email address so we can get in touch with you." }, "mode": { "title": "Mode", "enum": [ "catalogue_browse", "dataset_pull" ], "type": "string", "description": "" }, "datasetIds": { "title": "Dataset IDs (for dataset_pull mode)", "type": "array", "description": "Canonical IDs from the data.gov.my catalogue (e.g. 'fuelprice', 'cpi_headline'). Leave empty for catalogue browse mode.", "items": { "type": "string" } }, "startDate": { "title": "Start Date (YYYY-MM-DD)", "type": "string", "description": "Filter observations on or after this date." }, "endDate": { "title": "End Date (YYYY-MM-DD)", "type": "string", "description": "Filter observations on or before this date." }, "publisherFilter": { "title": "Publisher Filter (catalogue mode only)", "enum": [ "any", "DOSM", "MOF", "MOH", "KKMM", "BNM" ], "type": "string", "description": "" }, "maxItems": { "title": "Max Records", "type": "integer", "description": "Maximum number of records to return.", "default": 10 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```