OpenAI Vector Store Safe Sync

Synchronize selected fields from one Apify dataset into an existing OpenAI vector store using deterministic, self-contained generations, exact ownership attributes, and a durable mutation journal.

Quick start

1. Run the prefilled demo input. It writes one clearly labelled DEMO receipt and makes no Apify Dataset or OpenAI API call.
2. Select plan, choose the source dataset, and replace the demo placeholders with an existing OpenAI vector store ID and your BYOK OpenAI API key.
3. Choose the fields to index, a stable unique record key, and a syncId, then run the plan and review its RUN-SUMMARY.
4. Use append for a non-destructive first apply. Use sync only after reviewing and copying the exact planHash when cleanup is proposed.

The Actor is pay per usage: there is no developer surcharge. Apify platform usage and your own OpenAI API, File, and vector-store storage charges are billed separately. A changed apply uploads a complete candidate generation for concurrency safety; an unchanged run uploads nothing.

Why this Actor exists

Periodic full replacement has two costly failure modes: unchanged content is uploaded again, and a partial replacement can remove the last valid corpus. This Actor makes the safety boundary explicit:

1. Read and validate the complete selected source before remote mutation.
2. Assign stable record keys to fixed hash buckets.
3. Return unchanged when every rendered bucket hash still matches.
4. Otherwise upload and verify a complete candidate generation. It never references a File from the generation it may replace.
5. Activate the new logical manifest.
6. In sync, require the reviewed planHash, then detach only exact File IDs from the immutable base manifest after live ownership attributes still match.

The complete-generation choice is deliberate. Changed-shard reuse is cheaper, but without a compare-and-set state store two concurrent runs can each commit a manifest that reuses a different old File and then detach what the other run still references. Apify key-value storage is durable but does not provide that atomic fence. This v1 spends more on changed runs to preserve a complete candidate under that race.

The underlying OpenAI File object is never deleted by this Actor. A detach only removes the File from the selected vector store. Any detached File IDs are returned as explicit cleanup candidates.

Modes

• demo (Store prefill): emits one explicit apify-openai-vector-demo-v1 preview receipt without reading a dataset, opening Actor state, or calling OpenAI. No paid Actor event is charged; ordinary Apify platform usage may apply.
• plan (default): reads the dataset, target attachment metadata, and durable manifest; emits a content-free plan; performs no OpenAI mutation.
• append: when content changed, uploads and activates a complete generation while retaining the prior attachments.
• sync: uploads and activates a complete generation, then detaches the exact prior generation after all safety checks pass.

Begin with plan. Review blockers, uploadShards, detachFileIds, and sensitiveFieldWarnings. Copy its exact planHash into expectedPlanHash before a sync that will detach old attachments. Apply fails if the source or destination changed after review.

Input contract

Required inputs:

• vectorStoreId: an existing OpenAI vector store.
• openaiApiKey: BYOK secret. It is not copied to logs or outputs.
• datasetId: selected through Apify's dataset picker, or supplied by an Actor integration payload.
• fields: an explicit allowlist of simple dot-separated object paths.
• recordKeyField: stable, scalar, present, and unique for every row.
• syncId: stable namespace for this dataset and configuration.

expectedPlanHash is required by sync whenever the plan contains cleanup targets. It is intentionally not required for a first generation with nothing to detach, or for non-destructive append.

The source dataset ID, shard count, selected fields, and record-key field are a semantic configuration. Use a new syncId instead of silently changing them for an existing manifest.

Hard limits fail the run before any OpenAI write. The Actor never truncates an oversized dataset or shard to make it fit.

State and concurrency

The Actor keeps versioned manifests and mutation journals in an Actor-owned named key-value store. This storage provides durability and resume evidence; it is not an atomic distributed lock.

Safety does not rely on a filename prefix or that journal acting as a lock. Every detach target must be both:

• an exact File ID in the operation's immutable base manifest; and
• a live completed attachment carrying the expected ownership, scope, shard, content hash, generation, and operation attributes.

Every changed apply is self-contained, so overlapping runs can leave more than one complete candidate attached without making the final manifest reference a detached base File. The Actor prefers retained complete duplicates and a failed/partial receipt over speculative cleanup. Do not intentionally schedule overlapping sync runs for the same vectorStoreId and syncId.

If cleanup is interrupted after manifest activation, the next sync resumes the active operation's exact persisted detach references before it can report an unchanged result or start another generation. Missing uploaded candidate Files are re-created before attachment.

Outputs

• Default dataset: one content-free receipt per reuse, upload, attach, detach, or retained cleanup candidate.
• RUN-SUMMARY: the single canonical receipt with full counts, plan hashes, File IDs, blockers, and outcome for automation and alerts.

Source field values and record keys are rendered into OpenAI shard Files but are not copied into Actor output records.

The Actor first persists a non-success RUN-SUMMARY, then detailed rows, and only then replaces the summary with a terminal success. Partial and failed results are persisted before the Actor marks the platform run failed. A single canonical record avoids contradictory terminal receipts because Apify KVS does not offer a multi-record transaction. SUCCEEDED therefore means the requested plan or apply operation reached its declared terminal state; it does not prove retrieval quality in a downstream application.

Retention and privacy

OpenAI states that /v1/files and /v1/vector_stores retain application state until deleted and are not eligible for Zero Data Retention. Select only fields you are authorized to send to OpenAI. Field names commonly associated with credentials or personal data produce a plan warning, but authorization and data minimization remain the caller's responsibility.

Optional fileExpiresAfterDays applies a 1-30 day expiry to newly uploaded Files, matching the OpenAI Files API bound. It is off by default because an expired File can invalidate a manifest if the caller does not refresh it in time. The Actor never changes the destination vector store's expiry.

References:

• OpenAI Retrieval guide
• OpenAI Files API
• OpenAI data controls
• Apify limited Actor permissions

Local development

python3 -m venv .venv
.venv/bin/python -m pip install -r requirements-dev.txt
.venv/bin/python -m pytest -q
.venv/bin/python -m compileall -q src
.venv/bin/ruff check src test_*.py scripts
.venv/bin/ruff format --check src test_*.py scripts
.venv/bin/mypy src

Regenerate persisted-contract schemas and the hash lock deliberately:

.venv/bin/python scripts/generate_contracts.py
sh scripts/lock_dependencies.sh

No deployment or live OpenAI canary is part of these local commands.