Multi-Cloud Apify Dataset Exporter — S3, Cloudflare R2, Backblaze B2 & Wasabi

Export any Apify dataset into your own storage, without building an ETL service. Upload compressed, partitioned, independently checksummed JSONL or CSV files to AWS S3, Cloudflare R2, Backblaze B2, Wasabi, or any trusted S3-compatible HTTPS endpoint. Every successful run creates a manifest with the exact bucket keys, row counts and SHA-256 checksums.

Publisher: OptiRefine · version: 0.1.0 (testable MVP) · pricing: one pay-per-event charge per successfully uploaded data part (proposed $0.02). A manifest or a dry run is not charged. The provider may charge separate storage and request fees.

This Actor is independent of AWS, Cloudflare, Backblaze, Wasabi and Apify. No third-party endorsement is implied.

Why this Actor exists

The official apify/s3-uploader supports AWS S3 and a single export object; this Actor focuses on S3-compatible storage beyond AWS, deterministic multi-part exports, per-part checksums, and incremental append-only dataset sync. It is not a generic scraper and does not need browser proxies or AI API keys.

Features

• Five destination options: Amazon S3, Cloudflare R2, Backblaze B2, Wasabi, and custom S3-compatible HTTPS.
• Source: Apify dataset ID, run ID (resolves to its default dataset) or inline sample rows for testing.
• Output: jsonl or csv, optional gzip; automatic partitioning by row limit and uncompressed byte limit.
• Dataset-to-cloud traceability: SHA-256 per uploaded file, row counts, file size, bucket and object key, manifest.json in the bucket and Apify's MANIFEST key-value-store record.
• Incremental mode: for stable append-only named Apify datasets, stores the committed row offset for the next scheduled run.
• Privacy: destination keys are marked as Apify secret inputs; the Actor never outputs the secret values, signs uploads directly, and does not transmit rows to any provider other than your configured storage.
• Demo: first run works in dryRun=true mode without bucket credentials; it builds a free, realistic manifest preview.
• Limits: 1000 parts max, 5 MiB uncompressed per part (configurable downwards), 1 million items hard maximum. No silently truncated CSV columns: column paths are required.

Test it without a cloud account

1. Create an Actor from the contents of this folder, or use the Apify CLI instructions below.
2. Keep the default dryRun: true, blank datasetId, and default sampleItems. Run it.
3. Examine Output (OUTPUT), MANIFEST in key-value store, and the dataset's part rows. dryRun is true; nothing has been uploaded, and no export events have been charged.

Real AWS S3 export

Start from samples/s3-example.json and replace dataset ID, bucket and region, then paste S3 credentials into Apify's secret fields (do not commit actual credentials in Git). Set dryRun to false. Use an IAM policy restricted to the destination bucket and prefix; prefer an access key that can only s3:PutObject and cannot list or delete data.

Cloudflare R2 export

Start from samples/r2-example.json. Paste the Cloudflare account ID (32 hex characters), R2 bucket name, and R2-specific S3 API token's access key and secret key. This Actor uses https://<account-id>.r2.cloudflarestorage.com and region=auto. Do not use your Cloudflare global API key here: create an R2 API token with object-write permission scoped to the chosen bucket.

For Backblaze B2 choose provider=backblaze, with the API's region (e.g. us-west-004) and S3-compatible application keys. For Wasabi choose provider=wasabi and a matching region (e.g. us-east-1). For MinIO/other compatible stores use provider=custom, a trusted public HTTPS DNS endpoint and forcePathStyle=true. On-premises/local IP endpoints are intentionally rejected. Private network connections are outside this MVP's scope.

Run automatically after a scraper finishes

Create a task for this Actor, set the source dataset field to the parent run's default dataset ({{resource.defaultDatasetId}}), then configure a successful-run integration or webhook workflow so the export task starts after the upstream Actor's run succeeds. A task parameter may need to be supplied through a custom integration template; the exact UI differs by account.

For an append-only named dataset that all runs write into, enable incremental and choose a unique cursorStoreName. The first export starts at 0 and saves the next offset after the manifest successfully uploads. Later runs send only rows appended since the prior committed offset. Do not use the same cursor name for multiple concurrent runs, use the same dataset each time, and do not rewrite/delete old rows in an append-only dataset.

Input reference

Option	Meaning
datasetId	Existing dataset ID; preferred source
sourceRunId	Alternative; resolves a past Actor run's default dataset
dryRun	Preview only, default true, no bucket credentials needed
provider	aws, r2, backblaze, wasabi, custom
bucket, region, r2AccountId, customEndpoint	Storage destination
accessKeyId, secretAccessKey	Destination-scoped credentials, secret inputs
keyPrefix	Relative bucket key prefix; default apify-exports
format, compressGzip	jsonl/csv, gzip defaults on
csvColumns	Required in CSV mode: stable columns, supports address.city
includeFields, omitFields	JSONL field selection and omission with dot paths
maxRowsPerPart, maxBytesPerPart, maxItems	Upload/compute limits
incremental, cursorStoreName	Append-only source cursor

Result example

{
  "ok": true,
  "dryRun": false,
  "provider": "r2",
  "bucket": "my-r2-bucket",
  "sourceDatasetId": "DATASET_ID",
  "exportedRows": 2000,
  "parts": 2,
  "manifestObjectKey": "apify-exports/DATASET_ID/2026-10-29T12-00-00-000Z/manifest.json"
}

Output is a description of the export, not a copy of every data row. Use the dataset's objectKey field and the manifest to load the objects from your own S3-compatible bucket. The default destination is private, so it does not publish download URLs.

Pricing & margins

Proposed billing event: part-exported at $0.02 per successful part (maximum 5 MiB uncompressed or maxRowsPerPart rows, whichever comes first). A run with 2000 rows and two parts is $0.04 of event charges; Apify's infrastructure is paid from publisher revenue according to the PPE model. Small exports cost one part. An empty or dry run costs no export-event fee.

Apify currently returns 80% of event revenue to publishers before the Actor's platform costs; a $0.04 export would contribute $0.032 before compute, Apify dataset reads, and outbound transfer. This is a pricing hypothesis, not measured operating profit. Validate with 1 MB, 100 MB and 1 GB live runs before committing to a retail price. The separate bucket provider may charge customers for its storage and request operations. The pricing model and actual event price must be configured in Apify Console's Publication tab; .actor/pay_per_event.json provides suggested event definitions.

Reliability and limitations

• A network or destination error fails the Actor; previously uploaded parts can remain in your bucket. The cursor is committed only after completing the run and uploading its manifest; a rerun can create duplicates under a new timestamp prefix. It does not claim exactly-once delivery.
• The maximum charge cap can stop the Actor after a data part has been uploaded. The partial export is reported, the cursor stays unchanged, and already uploaded files remain in the bucket. The manifest records the partial result.
• Empty datasets produce no uploaded data parts or manifest object; the Apify OUTPUT/MANIFEST records still describe the empty result.
• The Actor does not encrypt data itself: destination transport uses HTTPS and encryption at rest is controlled by your bucket settings. Never send personal or regulated data without checking your obligations.
• CSV formula-like untrusted string values are prefixed with ' to reduce spreadsheet formula injection; numerical negative values remain numerical.
• It does not support Parquet, Azure Blob, Google Cloud Storage's native JSON API, BigQuery, automatic schema inference for CSV, active export retry checkpoints, or bucket credential auto-provisioning.
• Compressed output parts are limited by uncompressed byte count. If one row alone exceeds the per-part limit the run fails explicitly rather than truncating it.

Local development

npm install
npm test
npm run check

# Requires the Apify CLI, login and a valid production Actor configuration
npm install -g apify-cli
apify login
apify push

The npm test suite covers JSONL/GZIP, partition limits, CSV escaping, path handling, pagination and run/dataset resolution without making remote calls. A full end-to-end cloud upload still requires live Apify and destination credentials and has not been performed by this package's authoring environment.

SEO / Store listing metadata

Title: Multi-Cloud Dataset Exporter — S3, Cloudflare R2, B2, Wasabi

Short description: Export Apify datasets and Actor runs to AWS S3, Cloudflare R2, Backblaze B2, Wasabi and MinIO-compatible storage. JSONL/CSV, gzip, change-only append mode, manifests, SHA-256. No browser or proxy.

Target keywords: apify to cloudflare r2, apify dataset to s3, apify data exporter, backblaze b2 apify, wasabi s3 apify, apify data pipeline, apify etl, export apify dataset jsonl, incremental dataset export, cloud storage integrations.

License

MIT — see LICENSE.