# Apify Dataset to OpenAI Vector Store Sync (`nomad-agent/openai-vector-store-safe-sync`) Actor

Safely sync selected fields from any Apify Dataset to an existing OpenAI Vector Store. Preview changes before writing, use deterministic sharding and resumable manifests, verify every replacement, and detach only Actor-owned stale attachments—never delete global OpenAI Files.

- **URL**: https://apify.com/nomad-agent/openai-vector-store-safe-sync.md
- **Developed by:** [Nomad Dev](https://apify.com/nomad-agent) (community)
- **Categories:**
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## OpenAI Vector Store Safe Sync

Synchronize selected fields from one Apify dataset into an existing OpenAI
vector store using deterministic, self-contained generations, exact ownership
attributes, and a durable mutation journal.

### Quick start

1. Run the prefilled `demo` input. It writes one clearly labelled `DEMO` receipt
   and makes no Apify Dataset or OpenAI API call.
2. Select `plan`, choose the source dataset, and replace the demo placeholders
   with an existing OpenAI vector store ID and your BYOK OpenAI API key.
3. Choose the fields to index, a stable unique record key, and a `syncId`, then
   run the plan and review its `RUN-SUMMARY`.
4. Use `append` for a non-destructive first apply. Use `sync` only after
   reviewing and copying the exact `planHash` when cleanup is proposed.

The Actor is pay per usage: there is no developer surcharge. Apify platform
usage and your own OpenAI API, File, and vector-store storage charges are billed
separately. A changed apply uploads a complete candidate generation for
concurrency safety; an unchanged run uploads nothing.

### Why this Actor exists

Periodic full replacement has two costly failure modes: unchanged content is
uploaded again, and a partial replacement can remove the last valid corpus.
This Actor makes the safety boundary explicit:

1. Read and validate the complete selected source before remote mutation.
2. Assign stable record keys to fixed hash buckets.
3. Return unchanged when every rendered bucket hash still matches.
4. Otherwise upload and verify a complete candidate generation. It never
   references a File from the generation it may replace.
5. Activate the new logical manifest.
6. In `sync`, require the reviewed `planHash`, then detach only exact File IDs
   from the immutable base manifest after live ownership attributes still
   match.

The complete-generation choice is deliberate. Changed-shard reuse is cheaper,
but without a compare-and-set state store two concurrent runs can each commit
a manifest that reuses a different old File and then detach what the other run
still references. Apify key-value storage is durable but does not provide that
atomic fence. This v1 spends more on changed runs to preserve a complete
candidate under that race.

The underlying OpenAI File object is never deleted by this Actor. A detach only
removes the File from the selected vector store. Any detached File IDs are
returned as explicit cleanup candidates.

### Modes

- `demo` (Store prefill): emits one explicit `apify-openai-vector-demo-v1`
  preview receipt without reading a dataset, opening Actor state, or calling
  OpenAI. No paid Actor event is charged; ordinary Apify platform usage may
  apply.
- `plan` (default): reads the dataset, target attachment metadata, and durable
  manifest; emits a content-free plan; performs no OpenAI mutation.
- `append`: when content changed, uploads and activates a complete generation
  while retaining the prior attachments.
- `sync`: uploads and activates a complete generation, then detaches the exact
  prior generation after all safety checks pass.

Begin with `plan`. Review `blockers`, `uploadShards`, `detachFileIds`, and
`sensitiveFieldWarnings`. Copy its exact `planHash` into `expectedPlanHash`
before a `sync` that will detach old attachments. Apply fails if the source or
destination changed after review.

### Input contract

Required inputs:

- `vectorStoreId`: an existing OpenAI vector store.
- `openaiApiKey`: BYOK secret. It is not copied to logs or outputs.
- `datasetId`: selected through Apify's dataset picker, or supplied by an
  Actor integration payload.
- `fields`: an explicit allowlist of simple dot-separated object paths.
- `recordKeyField`: stable, scalar, present, and unique for every row.
- `syncId`: stable namespace for this dataset and configuration.

`expectedPlanHash` is required by `sync` whenever the plan contains cleanup
targets. It is intentionally not required for a first generation with nothing
to detach, or for non-destructive `append`.

The source dataset ID, shard count, selected fields, and record-key field are a
semantic configuration. Use a new `syncId` instead of silently changing them
for an existing manifest.

Hard limits fail the run before any OpenAI write. The Actor never truncates an
oversized dataset or shard to make it fit.

### State and concurrency

The Actor keeps versioned manifests and mutation journals in an Actor-owned
named key-value store. This storage provides durability and resume evidence;
it is not an atomic distributed lock.

Safety does not rely on a filename prefix or that journal acting as a lock.
Every detach target must be both:

- an exact File ID in the operation's immutable base manifest; and
- a live completed attachment carrying the expected ownership, scope, shard,
  content hash, generation, and operation attributes.

Every changed apply is self-contained, so overlapping runs can leave more than
one complete candidate attached without making the final manifest reference a
detached base File. The Actor prefers retained complete duplicates and a
failed/partial receipt over speculative cleanup. Do not intentionally schedule
overlapping `sync` runs for the same `vectorStoreId` and `syncId`.

If cleanup is interrupted after manifest activation, the next `sync` resumes
the active operation's exact persisted detach references before it can report
an unchanged result or start another generation. Missing uploaded candidate
Files are re-created before attachment.

### Outputs

- Default dataset: one content-free receipt per reuse, upload, attach, detach,
  or retained cleanup candidate.
- `RUN-SUMMARY`: the single canonical receipt with full counts, plan hashes,
  File IDs, blockers, and outcome for automation and alerts.

Source field values and record keys are rendered into OpenAI shard Files but
are not copied into Actor output records.

The Actor first persists a non-success `RUN-SUMMARY`, then detailed rows, and
only then replaces the summary with a terminal success. Partial and failed
results are persisted before the Actor marks the platform run failed. A single
canonical record avoids contradictory terminal receipts because Apify KVS does
not offer a multi-record transaction. `SUCCEEDED` therefore means the requested
plan or apply operation reached its declared terminal state; it does not prove
retrieval quality in a downstream application.

### Retention and privacy

OpenAI states that `/v1/files` and `/v1/vector_stores` retain application state
until deleted and are not eligible for Zero Data Retention. Select only fields
you are authorized to send to OpenAI. Field names commonly associated with
credentials or personal data produce a plan warning, but authorization and
data minimization remain the caller's responsibility.

Optional `fileExpiresAfterDays` applies a 1-30 day expiry to newly uploaded
Files, matching the OpenAI Files API bound. It is off by default because an
expired File can invalidate a manifest if the caller does not refresh it in
time. The Actor never changes the destination vector store's expiry.

References:

- [OpenAI Retrieval guide](https://developers.openai.com/api/docs/guides/retrieval)
- [OpenAI Files API](https://developers.openai.com/api/reference/resources/files)
- [OpenAI data controls](https://developers.openai.com/api/docs/guides/your-data)
- [Apify limited Actor permissions](https://docs.apify.com/actors/running/permissions)

### Local development

```bash
python3 -m venv .venv
.venv/bin/python -m pip install -r requirements-dev.txt
.venv/bin/python -m pytest -q
.venv/bin/python -m compileall -q src
.venv/bin/ruff check src test_*.py scripts
.venv/bin/ruff format --check src test_*.py scripts
.venv/bin/mypy src
```

Regenerate persisted-contract schemas and the hash lock deliberately:

```bash
.venv/bin/python scripts/generate_contracts.py
sh scripts/lock_dependencies.sh
```

No deployment or live OpenAI canary is part of these local commands.

# Actor input Schema

## `mode` (type: `string`):

Demo makes no external calls. Plan inspects real inputs without OpenAI writes. Append retains old attachments. Safe sync detaches only the reviewed prior generation.

## `vectorStoreId` (type: `string`):

Existing destination vector store. Demo ignores the prefilled placeholder; replace it before Plan, Append, or Sync. This Actor never creates, deletes, renames, or changes the store expiry.

## `openaiApiKey` (type: `string`):

BYOK key with permission to read the destination vector store and create/attach/detach Files. Demo ignores the placeholder; replace it before a real run. The key is never written to outputs or logs.

## `datasetId` (type: `string`):

Required for manual Plan, Append, and Sync runs; integrations may supply it automatically. Demo does not read a dataset. The Actor requests read-only access.

## `fields` (type: `array`):

Explicit allowlist of simple dot-separated dataset paths. Only these values and the record key are rendered into shard files.

## `recordKeyField` (type: `string`):

Simple dot-separated path whose value is present, scalar, stable, and unique for every row, such as id or url.

## `syncId` (type: `string`):

Stable namespace for this dataset/configuration within the vector store. Demo ignores the placeholder; replace it for real runs. Use a new value if fields, key, or shard count changes.

## `expectedPlanHash` (type: `string`):

Required by Sync when old attachments will be detached. Copy the exact planHash from a reviewed Plan run; apply fails if source or destination changed.

## `shardCount` (type: `integer`):

Persistent bucket count. It bounds File size and the number of Files in each complete generation. Do not change it for an existing syncId.

## `maxRows` (type: `integer`):

Fail without OpenAI writes if the source exceeds this count; data is never silently truncated.

## `maxProjectedBytes` (type: `integer`):

Fail before OpenAI writes when selected record keys and field values exceed this bound.

## `maxShardBytes` (type: `integer`):

Hard rendered-file bound applied before any OpenAI write.

## `maxUploadShards` (type: `integer`):

Fail before OpenAI writes if a complete candidate generation would upload more buckets. Values above shardCount simply allow every bucket.

## `maxDetachFiles` (type: `integer`):

Maximum exact base-manifest attachments a sync run may detach. Raise only after reviewing a Plan result.

## `fileExpiresAfterDays` (type: `integer`):

Optional OpenAI File expiry from creation. Leave empty unless the synchronization schedule refreshes content before expiry.

## `pollIntervalMs` (type: `integer`):

OpenAI vector-store batch polling interval.

## Actor input object example

```json
{
  "mode": "demo",
  "vectorStoreId": "vs_demo_not_used",
  "openaiApiKey": "demo-not-used",
  "fields": [
    "url",
    "text"
  ],
  "recordKeyField": "id",
  "syncId": "first-run-preview",
  "shardCount": 64,
  "maxRows": 50000,
  "maxProjectedBytes": 100000000,
  "maxShardBytes": 4000000,
  "maxUploadShards": 64,
  "maxDetachFiles": 64,
  "pollIntervalMs": 1000
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

## `runSummary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "mode": "demo",
    "vectorStoreId": "vs_demo_not_used",
    "openaiApiKey": "demo-not-used",
    "fields": [
        "url",
        "text"
    ],
    "recordKeyField": "id",
    "syncId": "first-run-preview"
};

// Run the Actor and wait for it to finish
const run = await client.actor("nomad-agent/openai-vector-store-safe-sync").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "mode": "demo",
    "vectorStoreId": "vs_demo_not_used",
    "openaiApiKey": "demo-not-used",
    "fields": [
        "url",
        "text",
    ],
    "recordKeyField": "id",
    "syncId": "first-run-preview",
}

# Run the Actor and wait for it to finish
run = client.actor("nomad-agent/openai-vector-store-safe-sync").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "mode": "demo",
  "vectorStoreId": "vs_demo_not_used",
  "openaiApiKey": "demo-not-used",
  "fields": [
    "url",
    "text"
  ],
  "recordKeyField": "id",
  "syncId": "first-run-preview"
}' |
apify call nomad-agent/openai-vector-store-safe-sync --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,nomad-agent/openai-vector-store-safe-sync"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/w2xkn3NZ83xeyBJKC/builds/voYyK58c2pRP1CGZA/openapi.json