Cloudflare R2 uploader

Uploads items from an apify dataset to a Cloudflare R2 bucket. This actor is not intended for general use. it is a building block for integrations.

features

• uploads each dataset item to R2 (dataset-only)
• integration-ready via dataset trigger
• dynamic overrides via input: endpoint url, bucket name, and secret credentials
• automatic retry logic with exponential backoff for transient failures
• logs uploaded keys and run metadata

setup

Requirements:

• Cloudflare R2 bucket
• R2 accessKeyId and secretAccessKey with write permissions

Local development requirements

env vars (local or platform) defined in .env.example file. Copy as .env and fill to test locally.

usage

• dataset trigger (local)
  • APIFY_INPUT='{"payload":{"datasetId":"DATASET-ID"}}' npm run start:dev --silent
• docker
  • docker build -t r2-uploader .
  • docker run --rm -e CF_R2_ENDPOINT -e CF_R2_ACCESS_KEY_ID -e CF_R2_SECRET_ACCESS_KEY -e CF_R2_BUCKET -e APIFY_INPUT='{"payload":{"datasetId":"DATASET-ID"}}' r2-uploader
• overrides via input
  • endpointUrl and bucketName override env vars
  • apiKey and secretKey override credential env vars

input parameters

• apiKey (string, secret)
  • cloudflare R2 access key id (overrides CF_R2_ACCESS_KEY_ID)
• secretKey (string, secret)
  • cloudflare R2 secret access key (overrides CF_R2_SECRET_ACCESS_KEY)
• endpointUrl (string)
  • cloudflare R2 endpoint url (overrides CF_R2_ENDPOINT)
  • must be a valid HTTPS URL for security
• bucketName (string)
  • cloudflare R2 bucket name (overrides CF_R2_BUCKET)
• filenamePattern (string, optional)
  • pattern for generating filenames. supports placeholders: {index}, {key}, {timestamp}
  • default: "dataset-item-{index}"
  • examples: "data-{timestamp}-{index}.json", "uploads/{key}", "batch-{index}"
• folderPath (string, optional)
  • folder path/prefix for organizing files in R2. supports {timestamp} placeholder
  • default: "" (root of bucket)
  • examples: "uploads/", "data/2024/", "datasets/{timestamp}/", "backups/daily/"
• conflictResolution (string, optional)
  • how to handle existing files with the same name
  • default: "overwrite"
  • options: "overwrite" (replace existing), "skip" (keep existing), "error" (fail on conflict)
• payload (object)
  • implicit payload used by apify integrations
  • datasetId (string): required in this actor to process dataset items

example inputs

{
    "endpointUrl": "https://31723x12x71x17271x12.r2.cloudflarestorage.com",
    "bucketName": "my-bucket",
    "filenamePattern": "item-{index}.json",
    "folderPath": "uploads/{timestamp}/",
    "datasetId": "YOUR-DATASET-ID"
}

This will create files like: uploads/2024-01-19T10:30:00.000Z/item-0.json

how it works

• dataset-trigger mode only:
  • payload.datasetId is required; the actor fetches dataset items and uploads each one
  • key resolution: item.key (if present) -> folderPath + filenamePattern with placeholders -> default pattern
  • conflict resolution: checks if files exist and handles based on strategy (overwrite/skip/error)
  • folder path: optional prefix with {timestamp} placeholder for organizing files
  • filename pattern placeholders:
    • {index}: item index (0, 1, 2, ...)
    • {key}: item.key value or fallback
    • {timestamp}: unique ISO timestamp per item (incremented by 1ms per item to ensure uniqueness)
  • body resolution: item.body -> base64(item.bodyBase64) -> JSON.stringify(item)
  • content type: text/plain for item.body, application/octet-stream for base64, application/json for stringified items

retry logic

• S3 upload operations automatically retry on transient failures
• exponential backoff strategy: 1s → 2s → 4s (with jitter)
• maximum 3 attempts per upload
• non-retryable errors (auth, permissions) fail immediately
• retry attempts are logged with details for debugging

examples

• dataset trigger input

{
  "endpointUrl": "https://<accountid>.R2.cloudflarestorage.com",
  "bucketName": "my-bucket",
  "filenamePattern": "data-{timestamp}-{index}.json",
  "folderPath": "uploads/{timestamp}/",
  "conflictResolution": "skip",
  "payload": { "datasetId": "YOUR-DATASET-ID" }
}

• dataset item expectations

items may include optional fields:

{
  "key": "path/to/object.json",
  "body": "string content",
  "bodyBase64": "..."
}

when body/bodyBase64 are missing, the whole item is stringified and uploaded.

error handling

• invalid credentials or endpoint -> authorization errors
• wrong bucket name or missing permissions -> access denied
• nonexistent or empty dataset -> warning and/or failure depending on input
• missing key/body in single-item mode -> validation error
• transient S3 failures -> automatic retry with exponential backoff (up to 3 attempts)

support

issues and ideas are welcome. if you need help, open an issue or contact us via apify support.