GCP Uploader

Google Cloud Storage Uploader

Apify Actor that uploads data from Apify datasets and Key-value stores directly to Google Cloud Storage (GCS).

Input

Required

key

Service account key in JSON format. Generate this in the Google Cloud Console for your service account. Stored as a secret.

{"type":"service_account","project_id":"my-project","private_key_id":"abc123","private_key":"-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----\n","client_email":"my-service-account@my-project.iam.gserviceaccount.com","client_id":"123456789","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token"}

bucketName

Name of the GCS bucket to upload files into.

---

Dataset upload

datasetId

ID of the Apify dataset to upload.

datasetFormat

Format in which the dataset is exported. Options: json (default), jsonl, xml, html, csv, xlsx, rss.

fileName

Name of the uploaded file. If omitted, the dataset ID is used as the file name.

deleteDatasetAfterUpload

Boolean (default false). When enabled, the source dataset is deleted from Apify after it has been successfully uploaded to the bucket. Nothing is deleted during a dry run.

---

Key-value store upload

kvStoreId

ID of the Apify Key-value store to upload records from.

kvStoreRegex

Optional regex. Only records whose key matches this pattern will be uploaded.

transformFileNameFunction

Optional JavaScript function to transform the record key into the destination file name. Receives the record key as its only argument and must return the new name.

(recordKey) => recordKey;

deleteRecordAfterUpload

Boolean (default false). When enabled, each Key-value store record is deleted from Apify right after it has been successfully uploaded to the bucket. Records skipped by kvStoreRegex are left untouched, and nothing is deleted during a dry run.

---

Additional options

uploadPath

Path prefix inside the bucket. Files are placed at <uploadPath>/<fileName>. If omitted, files are uploaded to the bucket root.

Example: my-folder/my-subfolder

compression

Compress files before uploading. Options: none (default), gzip, zip. The appropriate extension (.gz / .zip) is appended automatically.

uploadMetadata

Boolean (default true). When enabled, sets the contentType metadata on each uploaded GCS file.

storeEntriesToDataset

Boolean (default true). When enabled, each uploaded file's metadata (fileName, contentType, fullUploadPath, etc.) is pushed to the Actor's default dataset.

maxConcurrency

Maximum number of files uploaded in parallel (default 23). Lower this value if you hit GCP rate limits.

uploadDelayMs

Milliseconds to wait before starting each file upload (default 0). Useful for throttling when uploading a large number of small files.

---

Testing

dryRun

Boolean (default false). When enabled, the Actor runs through all logic but skips the actual GCS upload. Use this to validate your configuration without transferring any data.