Encrypted Data Integration

Encrypt structured records inside Apify before export, sync, or handoff. Load data from a dataset, key-value store, inline JSON records, or pasted JSON/CSV; apply AES-256-GCM (or Fernet) encryption; write ciphertext to a dataset, KV bundle, or both. Built for PII-safe pipelines, compliance-oriented workflows, and downstream automation without leaking plaintext into logs or exports.

Quick start

1. Open the Actor in Apify Console and click Start (defaults work for a demo run).
2. Leave Passphrase and Source JSON records empty to use built-in demo data (testing only).
3. Open the run dataset — encrypted rows (run totals are in KV store record RUN_SUMMARY.json).

Replace the demo passphrase before any production workload.

Usage

Console

1. Set Source mode (json_records for quick tests, dataset to encrypt another Actor’s output).
2. Choose Encryption scope and list Fields to encrypt for field-level mode.
3. Enter a strong Passphrase (secret input) or Raw key (base64).
4. Enable Remove plaintext after encryption when exporting sensitive columns.
5. Run and export results as JSON/CSV from the dataset, or read the KV bundle when Output mode is kv_store or both.

API / scheduler

Pass the same JSON as the input schema. Secret fields (passphrase, sourceJsonRecords, etc.) are encrypted at rest on Apify. Chain this Actor after scrapers via Scheduler or apify-client actor().call().

Example input

{
  "sourceMode": "json_records",
  "sourceJsonRecords": [
    { "id": "1", "email": "user@company.com", "notes": "Confidential memo" }
  ],
  "encryptionScope": "selected_fields",
  "fieldsToEncrypt": ["email", "notes"],
  "preserveFields": ["id"],
  "removePlaintextAfterEncryption": true,
  "algorithm": "aes_gcm",
  "keyMode": "passphrase",
  "passphrase": "USE_A_ROTATED_SECRET_FROM_A_VAULT",
  "keyDerivation": "pbkdf2_sha256",
  "iterations": 200000,
  "outputMode": "dataset",
  "includeManifest": true,
  "includeHashFingerprint": true
}

Encrypt from another dataset

{
  "sourceMode": "dataset",
  "sourceDatasetId": "YOUR_SOURCE_DATASET_ID",
  "maxItems": 250,
  "encryptionScope": "selected_fields",
  "fieldsToEncrypt": ["email", "phone"],
  "preserveFields": ["id", "name"],
  "algorithm": "aes_gcm",
  "keyMode": "passphrase",
  "passphrase": "USE_A_ROTATED_SECRET",
  "outputMode": "dataset"
}

Copy the Source dataset ID from the upstream Actor run’s default dataset URL in Apify Console.

Apify Console (health check & quality score)

After deploying build 0.6+:

1. Leave Passphrase and Source JSON records empty — the Actor applies built-in demo data with 1000 KDF iterations (finishes in under 1 minute).
2. Prefilled input uses sourceMode: json_records, maxItems: 10, maxConcurrency: 2, iterations: 1000.
3. Re-run Try actor with default prefilled input — expect encrypted demo rows in the dataset.

For production, set a strong passphrase (secret input) and raise iterations to 200000 or higher.

Authentication & sensitive input

Fields that hold credentials or record payloads use Apify secret input (isSecret: true):

• sourceJsonRecords — inline JSON records when sourceMode=json_records
• sourceJsonText — pasted JSON/CSV when sourceMode=json_text
• passphrase — encryption passphrase when keyMode=passphrase
• rawKeyBase64 — raw symmetric key when keyMode=raw_key_base64
• deterministicFingerprintSalt — optional fingerprint salt (not used for encryption)

Secret values are encrypted at rest and are not written into dataset rows or logs when redactLogs is enabled.

Input

Group	Main fields
Data source	sourceMode, sourceDatasetId, sourceKvStoreKey, sourceJsonRecords, sourceJsonText, maxItems
Encryption	encryptionScope, fieldsToEncrypt, preserveFields, removePlaintextAfterEncryption, algorithm, keyMode
Secrets	passphrase, rawKeyBase64, deterministicFingerprintSalt (isSecret)
Output	outputMode, outputKvStoreKey, includeManifest, includeHashFingerprint
Performance	chunkSize, maxConcurrency

Full schema: INPUT_SCHEMA.json.

Encryption modes

• selected_fields — Encrypt listed fields (dotted paths supported, e.g. contact.email).
• full_record — One ciphertext blob per JSON object.
• full_payload — One ciphertext blob for the entire batch.

Output

Each successful run writes to the default dataset (unless configured otherwise):

Row type	Meaning
Encrypted record	Plain object with *_encrypted fields and optional manifest / fingerprints
type: "__error__"	Per-record or configuration diagnostics (no secrets)
KV RUN_SUMMARY.json	Run totals: processedRecords, failedRecords, encryptedFieldsCount

Example encrypted field:

{
  "recordId": "1",
  "id": "1",
  "email_encrypted": {
    "algorithm": "aes_gcm",
    "version": "1",
    "nonce": "BASE64_NONCE",
    "salt": "BASE64_SALT",
    "kdf": "pbkdf2_sha256",
    "iterations": 200000,
    "ciphertext": "BASE64_CIPHERTEXT"
  },
  "manifest": {
    "cryptoVersion": "1",
    "encryptionScope": "selected_fields",
    "encryptedFields": ["email"]
  }
}

When outputMode is kv_store or both, a JSON bundle is stored under outputKvStoreKey (default ENCRYPTED_OUTPUT).

Pricing

This Actor supports Pay per event (PPE) on Apify Store:

Event	When charged
apify-actor-start	Each run start (platform-managed; optional free compute window)
record-encrypted	Each encrypted output row pushed to the dataset (primary value event)

Configure events and Bronze / Silver / Gold discounts in Console → Publication → Monetization. See PUBLISHING.md for a step-by-step checklist to reach a 100/100 quality score.

Without PPE, the Actor still runs under standard platform usage billing.

Security

• AES-256-GCM authenticated encryption; passphrases derived with PBKDF2 or scrypt.
• Sensitive inputs use Apify isSecret encryption at rest.
• redactLogs and removePlaintextAfterEncryption reduce accidental exposure.
• Fingerprints are SHA-256 hashes for matching — not confidentiality.
• Rotate passphrases and restrict dataset access in production.

FAQ

Question	Answer
First run with empty secrets?	Demo records + demo passphrase (fast KDF). Not for production.
CRM / Salesforce integration?	Encrypt here; sync ciphertext with your own exporters.
Missing fields?	__error__ rows unless failOnMissingFields is true.
KV input vs output?	sourceKvStoreKey and outputKvStoreKey must differ when both are used.

Local development

cd encrypted-data-integration
python -m pip install -r requirements.txt
python scripts/roundtrip_validation.py

Add INPUT.json for full local runs via python main.py.

Changelog

See CHANGELOG.md.

Publishing

See PUBLISHING.md for Console steps (PPE, SEO, categories, limited permissions, Store discounts).