# Export Dataset to Parquet - Typed Schema for DuckDB & BigQuery (`that_red_bird/dataset-to-parquet`) Actor

Apify exports JSON, CSV and Excel but not Parquet. Pick any dataset and get typed Parquet files (inferred schema, flattened nested fields, detected timestamps) in the key-value store, optionally pushed to S3, R2 or Google Cloud Storage. Ready for DuckDB, BigQuery, Athena.

- **URL**: https://apify.com/that_red_bird/dataset-to-parquet.md
- **Developed by:** [mohamed alaya](https://apify.com/that_red_bird) (community)
- **Categories:** Developer tools, Automation, Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$2.00 / 1,000 1,000 rows converteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Export Dataset to Parquet - Typed Schema for DuckDB & BigQuery

Converts any Apify dataset (or pasted JSON/CSV rows) into typed Parquet files, saved in the key-value store and optionally uploaded to Amazon S3, an S3-compatible service or Google Cloud Storage.
For data engineers and analysts who load scrapes into DuckDB, BigQuery, Athena or Spark and are tired of CSV type guessing: Apify exports JSON, CSV, Excel, XML and RSS, but not Parquet.

### Quick start

1. Run any scraper, then start this Actor and pick that run's dataset in **Dataset(s) to convert**.
2. Open the run's key-value store: `dataset.parquet` is there, with a `SCHEMA` record that lists every column and type.
3. Query it: `SELECT * FROM read_parquet('dataset.parquet')` in DuckDB, or load it into BigQuery / Athena / Spark.

Run with empty input to convert a demo of 1,500 nested order rows.

### Input examples

**1. Dataset from a scraper**

```json
{ "datasets": ["<pick with the dataset picker>"], "fileName": "google-maps-2026-10" }
```

**2. Pasted rows**

```json
{ "rows": [{ "id": 1, "customer": { "name": "Ann" }, "tags": ["a", "b"], "createdAt": "2026-05-01T10:00:00Z" }] }
```

**3. Dataset to Parquet files in S3 as well**

```json
{
  "datasets": ["<pick with the dataset picker>"],
  "s3Bucket": "my-data-lake", "s3Region": "eu-west-1", "s3KeyPrefix": "scrapes/",
  "s3AccessKeyId": "AKIA...", "s3SecretAccessKey": "..."
}
```

### Input guide

| Field | What it does |
|---|---|
| `datasets` | Pick one or more datasets; they are appended into one schema. |
| `rows`, `rowsCsv` | Paste JSON rows, a JSON array or CSV (numbers and true/false in CSV are detected). |
| `fileName` | Base name of the output file(s). |
| `nestedFields` | `flatten` (`a.b` becomes column `a_b`) or `json` (keep nested objects as one JSON column). Arrays are always JSON columns. |
| `inferDates` | Columns of ISO 8601 dates/date-times become TIMESTAMP (no time zone = UTC). |
| `sanitizeColumnNames` | Letters, digits and `_` only, unique names (needed for BigQuery and Athena). |
| `rowsPerFile` / `maxRows` | Split big datasets into several files / safety limit. |
| `s3*` | Optional S3 or S3-compatible upload (set `s3Endpoint` for Cloudflare R2, MinIO, Backblaze B2). |
| `gcs*` | Optional Google Cloud Storage upload with a service-account JSON key. |

Credentials are marked secret, so Apify stores them encrypted. Use a key that can only write to one bucket.

### Output

The dataset has one row per Parquet file:

| Field | Meaning |
|---|---|
| `fileName` | Key of the file in the run's key-value store. |
| `url` | API URL of the file (needs your Apify token or a logged-in browser). |
| `rows`, `columns`, `bytes` | Size of the file. |
| `schema` | Column name to Parquet type (`INT32`, `INT64`, `DOUBLE`, `BOOLEAN`, `TIMESTAMP`, `STRING`, `JSON`). |
| `uploads` | Result of each S3 / GCS upload: `ok` with the `location`, or the `error`. |

Type rules: a column with only integers is INT32 (INT64 if any value is larger), integers mixed with decimals become DOUBLE, true/false is BOOLEAN, ISO dates are TIMESTAMP, nested arrays are JSON, and a column that mixes incompatible types (for example numbers and text) is stored as STRING. Empty values become NULL.

Sample row:

```json
{ "type": "parquet-file", "fileName": "dataset.parquet", "rows": 1500, "columns": 9, "bytes": 40021, "schema": { "orderId": "INT32", "customer_name": "STRING", "total": "DOUBLE", "paid": "BOOLEAN", "createdAt": "TIMESTAMP", "tags": "JSON" }, "uploads": [] }
```

### Use it with

Any scraper -> **this Actor** -> DuckDB / BigQuery / Athena / Snowflake. Chain it with the Google Maps scraper and the other data-cleaning Actors from the same author (names, countries, hours) before exporting.

### FAQ

**Why not CSV?** CSV loses types (everything is text), mangles nested data and is 5 to 10 times bigger.

**Does it use DuckDB?** It uses a pure JavaScript Parquet writer (hyparquet-writer, Snappy compression), so there is no native build to break. The output was checked by reading it back with DuckDB.

**Is BigQuery fine with the JSON columns?** BigQuery loads them as STRING; DuckDB reads them as JSON. Use `nestedFields: flatten` and avoid arrays if you need pure scalar columns.

### Honest limits

- The S3 and GCS uploads are plain single-request uploads (fine up to roughly 1 to 2 GB per file; lower `rowsPerFile` for bigger data). They are built on the standard signing protocols but were not tested against every provider.
- All rows of one file are held in memory while it is written (the dataset itself is streamed). Keep `rowsPerFile` modest for very wide rows.
- Schemas are inferred from all rows; there is no way to force a type per column yet.

### Price

Pay per event: **$0.002 per 1,000 rows** written to Parquet (billed per started 1,000 rows of each file; failed files and the uploads are free). 100,000 rows = $0.20, 1 million rows = $2.

# Actor input Schema

## `datasets` (type: `array`):

Pick the dataset of any Actor run. Use the picker rather than typing an ID: it gives this Actor permission to read your dataset. Several datasets are appended into one Parquet schema. Leave empty to use the pasted rows below, or the built-in demo.

## `rows` (type: `array`):

Optional. Paste rows as JSON objects, e.g. {"id":1,"name":"Ann","tags":\["a"]}. Nested objects are flattened by default.

## `rowsCsv` (type: `string`):

Optional. Paste a CSV (first row = column names; numbers and true/false are detected) or a JSON array.

## `fileName` (type: `string`):

Base name of the Parquet file in the key-value store (.parquet is added). With several files the name gets -part-0001, -part-0002 and so on.

## `nestedFields` (type: `string`):

flatten turns {"a":{"b":1}} into a column a_b (best for BigQuery, Athena, DuckDB). json keeps each top-level nested object as one JSON column. Arrays are always kept as JSON columns.

## `inferDates` (type: `boolean`):

Columns where every value is an ISO 8601 date or date-time (2026-05-01, 2026-05-01T10:00:00Z) become TIMESTAMP columns. Date-times without a time zone are read as UTC.

## `sanitizeColumnNames` (type: `boolean`):

Replace characters other than letters, digits and underscore with an underscore (BigQuery and Athena require this) and make names unique. Turn off to keep the original names.

## `rowsPerFile` (type: `integer`):

Maximum rows in one Parquet file. Bigger datasets are split into several files with the same schema; each file is billed per started 1,000 rows.

## `maxRows` (type: `integer`):

Safety limit on the number of rows converted in one run.

## `s3Bucket` (type: `string`):

Also upload each Parquet file to this Amazon S3 bucket (or any S3-compatible service such as Cloudflare R2, MinIO or Backblaze B2). Needs the access key fields below. The files are always kept in the key-value store too.

## `s3Region` (type: `string`):

AWS region of the bucket, for example us-east-1 or eu-west-1. For R2 use auto.

## `s3KeyPrefix` (type: `string`):

Optional folder inside the bucket, for example scrapes/2026-10/.

## `s3AccessKeyId` (type: `string`):

Access key ID of an IAM user that may PutObject into the bucket. Use a key limited to this bucket.

## `s3SecretAccessKey` (type: `string`):

Secret access key that belongs to the access key ID. Stored encrypted by Apify.

## `s3SessionToken` (type: `string`):

Only for temporary credentials (AWS STS). Leave empty otherwise.

## `s3Endpoint` (type: `string`):

Only for non-AWS services, for example https://<account>.r2.cloudflarestorage.com or https://s3.us-west-004.backblazeb2.com. Leave empty for Amazon S3.

## `gcsBucket` (type: `string`):

Also upload each Parquet file to this Google Cloud Storage bucket. Needs the service-account JSON below.

## `gcsKeyPrefix` (type: `string`):

Optional folder inside the bucket, for example scrapes/2026-10/.

## `gcsServiceAccountJson` (type: `string`):

Paste the full JSON key of a service account with Storage Object Creator access to the bucket. Stored encrypted by Apify.

## Actor input object example

```json
{
  "rows": [],
  "rowsCsv": "",
  "fileName": "dataset",
  "nestedFields": "flatten",
  "inferDates": true,
  "sanitizeColumnNames": true,
  "rowsPerFile": 250000,
  "maxRows": 1000000,
  "s3Bucket": "",
  "s3Region": "us-east-1",
  "s3KeyPrefix": "",
  "s3Endpoint": "",
  "gcsBucket": "",
  "gcsKeyPrefix": ""
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `downloadCsv` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `count` (type: `string`):

No description

## `schemaRecord` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "rows": [],
    "rowsCsv": "",
    "fileName": "dataset",
    "nestedFields": "flatten",
    "rowsPerFile": 250000,
    "maxRows": 1000000,
    "s3Bucket": "",
    "s3Region": "us-east-1",
    "s3KeyPrefix": "",
    "s3Endpoint": "",
    "gcsBucket": "",
    "gcsKeyPrefix": ""
};

// Run the Actor and wait for it to finish
const run = await client.actor("that_red_bird/dataset-to-parquet").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "rows": [],
    "rowsCsv": "",
    "fileName": "dataset",
    "nestedFields": "flatten",
    "rowsPerFile": 250000,
    "maxRows": 1000000,
    "s3Bucket": "",
    "s3Region": "us-east-1",
    "s3KeyPrefix": "",
    "s3Endpoint": "",
    "gcsBucket": "",
    "gcsKeyPrefix": "",
}

# Run the Actor and wait for it to finish
run = client.actor("that_red_bird/dataset-to-parquet").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "rows": [],
  "rowsCsv": "",
  "fileName": "dataset",
  "nestedFields": "flatten",
  "rowsPerFile": 250000,
  "maxRows": 1000000,
  "s3Bucket": "",
  "s3Region": "us-east-1",
  "s3KeyPrefix": "",
  "s3Endpoint": "",
  "gcsBucket": "",
  "gcsKeyPrefix": ""
}' |
apify call that_red_bird/dataset-to-parquet --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,that_red_bird/dataset-to-parquet"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/AHfDsu9h8tlPBpaSo/builds/Ah1W5twmBTqhqWPT8/openapi.json