# Multi-Cloud Dataset Exporter — S3, R2, B2 & Wasabi (`optirefine/multicloud-dataset-exporter`) Actor

Send Apify datasets to AWS S3, Cloudflare R2, Backblaze B2, Wasabi or an S3-compatible bucket. Partitioned JSONL/CSV, gzip, manifest, and optional incremental export.

- **URL**: https://apify.com/optirefine/multicloud-dataset-exporter.md
- **Developed by:** [OptiRefine](https://apify.com/optirefine) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $20.00 / 1,000 dataset items

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Multi-Cloud Apify Dataset Exporter — S3, Cloudflare R2, Backblaze B2 & Wasabi

Export any Apify dataset into **your own storage**, without building an ETL service. Upload compressed, partitioned, independently checksummed JSONL or CSV files to **AWS S3, Cloudflare R2, Backblaze B2, Wasabi, or any trusted S3-compatible HTTPS endpoint**. Every successful run creates a manifest with the exact bucket keys, row counts and SHA-256 checksums.

**Publisher:** OptiRefine · **version:** 0.1.0 (testable MVP) · **pricing:** one pay-per-event charge per *successfully uploaded data part* (proposed $0.02). A manifest or a dry run is not charged. The provider may charge separate storage and request fees.

This Actor is independent of AWS, Cloudflare, Backblaze, Wasabi and Apify. No third-party endorsement is implied.

### Why this Actor exists

The official `apify/s3-uploader` supports AWS S3 and a single export object; this Actor focuses on **S3-compatible storage beyond AWS, deterministic multi-part exports, per-part checksums, and incremental append-only dataset sync**. It is not a generic scraper and does not need browser proxies or AI API keys.

#### Features

- **Five destination options:** Amazon S3, Cloudflare R2, Backblaze B2, Wasabi, and custom S3-compatible HTTPS.
- **Source:** Apify dataset ID, run ID (resolves to its default dataset) or inline sample rows for testing.
- **Output:** `jsonl` or `csv`, optional gzip; automatic partitioning by row limit and uncompressed byte limit.
- **Dataset-to-cloud traceability:** SHA-256 per uploaded file, row counts, file size, bucket and object key, `manifest.json` in the bucket and Apify's `MANIFEST` key-value-store record.
- **Incremental mode:** for stable append-only named Apify datasets, stores the committed row offset for the next scheduled run.
- **Privacy:** destination keys are marked as Apify secret inputs; the Actor never outputs the secret values, signs uploads directly, and does not transmit rows to any provider other than your configured storage.
- **Demo:** first run works in `dryRun=true` mode without bucket credentials; it builds a free, realistic manifest preview.
- **Limits:** 1000 parts max, 5 MiB uncompressed per part (configurable downwards), 1 million items hard maximum. No silently truncated CSV columns: column paths are required.

### Test it without a cloud account

1. Create an Actor from the contents of this folder, or use the Apify CLI instructions below.
2. Keep the default `dryRun: true`, blank `datasetId`, and default `sampleItems`. Run it.
3. Examine **Output** (`OUTPUT`), **MANIFEST** in key-value store, and the dataset's part rows. `dryRun` is `true`; nothing has been uploaded, and no export events have been charged.

### Real AWS S3 export

Start from `samples/s3-example.json` and replace **dataset ID, bucket and region**, then paste S3 credentials into Apify's secret fields (do not commit actual credentials in Git). Set `dryRun` to `false`. Use an IAM policy restricted to the destination bucket and prefix; prefer an access key that can only `s3:PutObject` and cannot list or delete data.

### Cloudflare R2 export

Start from `samples/r2-example.json`. Paste the Cloudflare **account ID** (32 hex characters), R2 bucket name, and R2-specific S3 API token's access key and secret key. This Actor uses `https://<account-id>.r2.cloudflarestorage.com` and `region=auto`. Do **not** use your Cloudflare global API key here: create an **R2 API token** with object-write permission scoped to the chosen bucket.

For Backblaze B2 choose `provider=backblaze`, with the API's region (e.g. `us-west-004`) and S3-compatible application keys. For Wasabi choose `provider=wasabi` and a matching region (e.g. `us-east-1`). For MinIO/other compatible stores use `provider=custom`, a **trusted public HTTPS DNS endpoint** and `forcePathStyle=true`. On-premises/local IP endpoints are intentionally rejected. Private network connections are outside this MVP's scope.

### Run automatically after a scraper finishes

Create a task for this Actor, set the source dataset field to the parent run's default dataset (`{{resource.defaultDatasetId}}`), then configure a successful-run integration or webhook workflow so the export task starts **after** the upstream Actor's run succeeds. A task parameter may need to be supplied through a custom integration template; the exact UI differs by account.

For an **append-only named dataset** that all runs write into, enable `incremental` and choose a unique `cursorStoreName`. The first export starts at 0 and saves the next offset after the manifest successfully uploads. Later runs send only rows appended since the prior committed offset. Do not use the same cursor name for multiple concurrent runs, use the same dataset each time, and do not rewrite/delete old rows in an append-only dataset.

### Input reference

| Option | Meaning |
|---|---|
| `datasetId` | Existing dataset ID; preferred source |
| `sourceRunId` | Alternative; resolves a past Actor run's default dataset |
| `dryRun` | Preview only, default `true`, no bucket credentials needed |
| `provider` | `aws`, `r2`, `backblaze`, `wasabi`, `custom` |
| `bucket`, `region`, `r2AccountId`, `customEndpoint` | Storage destination |
| `accessKeyId`, `secretAccessKey` | Destination-scoped credentials, secret inputs |
| `keyPrefix` | Relative bucket key prefix; default `apify-exports` |
| `format`, `compressGzip` | `jsonl`/`csv`, gzip defaults on |
| `csvColumns` | Required in CSV mode: stable columns, supports `address.city` |
| `includeFields`, `omitFields` | JSONL field selection and omission with dot paths |
| `maxRowsPerPart`, `maxBytesPerPart`, `maxItems` | Upload/compute limits |
| `incremental`, `cursorStoreName` | Append-only source cursor |

#### Result example

```json
{
  "ok": true,
  "dryRun": false,
  "provider": "r2",
  "bucket": "my-r2-bucket",
  "sourceDatasetId": "DATASET_ID",
  "exportedRows": 2000,
  "parts": 2,
  "manifestObjectKey": "apify-exports/DATASET_ID/2026-10-29T12-00-00-000Z/manifest.json"
}
```

**Output is a description of the export, not a copy of every data row.** Use the dataset's `objectKey` field and the manifest to load the objects from your own S3-compatible bucket. The default destination is private, so it does not publish download URLs.

### Pricing & margins

Proposed billing event: `part-exported` at **$0.02 per successful part** (maximum **5 MiB uncompressed** or `maxRowsPerPart` rows, whichever comes first). A run with 2000 rows and two parts is $0.04 of event charges; Apify's infrastructure is paid from publisher revenue according to the PPE model. Small exports cost one part. An empty or dry run costs no export-event fee.

Apify currently returns **80%** of event revenue to publishers *before* the Actor's platform costs; a $0.04 export would contribute $0.032 before compute, Apify dataset reads, and outbound transfer. This is a pricing hypothesis, **not measured operating profit**. Validate with 1 MB, 100 MB and 1 GB live runs before committing to a retail price. The separate bucket provider may charge customers for its storage and request operations. The pricing model and actual event price must be configured in Apify Console's Publication tab; `.actor/pay_per_event.json` provides suggested event definitions.

### Reliability and limitations

- A network or destination error fails the Actor; previously uploaded parts can remain in your bucket. The cursor is committed only after completing the run and uploading its manifest; a rerun can create duplicates under a new timestamp prefix. It does not claim exactly-once delivery.
- The maximum charge cap can stop the Actor after a data part has been uploaded. The partial export is reported, the cursor stays unchanged, and already uploaded files remain in the bucket. The manifest records the partial result.
- Empty datasets produce no uploaded data parts or manifest object; the Apify `OUTPUT`/`MANIFEST` records still describe the empty result.
- The Actor does not encrypt data itself: destination transport uses HTTPS and encryption at rest is controlled by your bucket settings. Never send personal or regulated data without checking your obligations.
- CSV formula-like untrusted string values are prefixed with `'` to reduce spreadsheet formula injection; numerical negative values remain numerical.
- It does not support Parquet, Azure Blob, Google Cloud Storage's native JSON API, BigQuery, automatic schema inference for CSV, active export retry checkpoints, or bucket credential auto-provisioning.
- Compressed output parts are limited by uncompressed byte count. If one row alone exceeds the per-part limit the run fails explicitly rather than truncating it.

### Local development

```bash
npm install
npm test
npm run check

## Requires the Apify CLI, login and a valid production Actor configuration
npm install -g apify-cli
apify login
apify push
```

The `npm test` suite covers JSONL/GZIP, partition limits, CSV escaping, path handling, pagination and run/dataset resolution without making remote calls. **A full end-to-end cloud upload still requires live Apify and destination credentials and has not been performed by this package's authoring environment.**

### SEO / Store listing metadata

**Title:** Multi-Cloud Dataset Exporter — S3, Cloudflare R2, B2, Wasabi

**Short description:** Export Apify datasets and Actor runs to AWS S3, Cloudflare R2, Backblaze B2, Wasabi and MinIO-compatible storage. JSONL/CSV, gzip, change-only append mode, manifests, SHA-256. No browser or proxy.

**Target keywords:** apify to cloudflare r2, apify dataset to s3, apify data exporter, backblaze b2 apify, wasabi s3 apify, apify data pipeline, apify etl, export apify dataset jsonl, incremental dataset export, cloud storage integrations.

### License

MIT — see `LICENSE`.

# Actor input Schema

## `provider` (type: `string`):

Select the cloud storage provider for your export.

## `bucket` (type: `string`):

Cloud storage bucket name. Required for a real upload; optional for dry-run previews.

## `region` (type: `string`):

Cloud storage region, such as us-east-1.

## `endpoint` (type: `string`):

Optional S3-compatible endpoint URL. Confirm the runtime supports this field name.

## `accessKeyId` (type: `string`):

Cloud storage access key identifier. Stored as encrypted input.

## `secretAccessKey` (type: `string`):

Cloud storage secret access key. Stored as encrypted input.

## `sessionToken` (type: `string`):

Optional temporary cloud authentication token. Stored as encrypted input.

## `datasetId` (type: `string`):

Select an Apify dataset to export. Required for incremental dataset processing.

## `runId` (type: `string`):

Optional source Actor run ID. Use only if supported by the exporter's runtime.

## `sampleItems` (type: `array`):

Sample records for a dry-run preview. Do not combine with incremental processing.

## `format` (type: `string`):

Output format supported by the exporter, such as jsonl.

## `keyPrefix` (type: `string`):

Prefix used when creating destination object keys.

## `maxItems` (type: `integer`):

Maximum number of source records to process during this run.

## `maxRowsPerPart` (type: `integer`):

Maximum number of records in each exported file.

## `maxBytesPerPart` (type: `integer`):

Maximum target size of each export part in bytes.

## `compressGzip` (type: `boolean`):

Compress exported data using GZIP.

## `forcePathStyle` (type: `boolean`):

Use path-style bucket addressing for compatible S3 endpoints.

## `incremental` (type: `boolean`):

Process new dataset records using saved export progress. Requires a real source dataset or run.

## `cursorStoreName` (type: `string`):

Persistent cursor storage identifier used for incremental export tracking.

## `dryRun` (type: `boolean`):

Generate an export preview without uploading files to cloud storage.

## Actor input object example

```json
{
  "provider": "r2",
  "region": "us-east-1",
  "sampleItems": [
    {
      "company": "OptiRefine",
      "sales": 500
    }
  ],
  "format": "jsonl",
  "keyPrefix": "apify-exports",
  "maxItems": 1,
  "maxRowsPerPart": 1000,
  "maxBytesPerPart": 5242880,
  "compressGzip": false,
  "forcePathStyle": true,
  "incremental": false,
  "cursorStoreName": "multicloud-exporter-cursor",
  "dryRun": true
}
```

# Actor output Schema

## `exportedParts` (type: `string`):

Table of parts generated or previewed by this run.

## `runRecords` (type: `string`):

List key-value store records; open OUTPUT for the run summary or MANIFEST for detailed file information.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sampleItems": [
        {
            "company": "OptiRefine",
            "sales": 500
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("optirefine/multicloud-dataset-exporter").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "sampleItems": [{
            "company": "OptiRefine",
            "sales": 500,
        }] }

# Run the Actor and wait for it to finish
run = client.actor("optirefine/multicloud-dataset-exporter").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sampleItems": [
    {
      "company": "OptiRefine",
      "sales": 500
    }
  ]
}' |
apify call optirefine/multicloud-dataset-exporter --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,optirefine/multicloud-dataset-exporter"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/172rsBMCqfwFECwLJ/builds/gNbjUBcJUWMELlk22/openapi.json