# Pinecone Dataset Sync & Repair (`vectorrelay/pinecone-dataset-sync-repair`) Actor

Sync Apify datasets to an existing Pinecone index and repair failed or inconsistent upserts. Uses Pinecone Integrated Embedding, so no OpenAI or Cohere API key is required.

- **URL**: https://apify.com/vectorrelay/pinecone-dataset-sync-repair.md
- **Developed by:** [VectorRelay](https://apify.com/vectorrelay) (community)
- **Categories:** AI, Developer tools, Integrations
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 record synced to pinecones

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## VectorRelay – Pinecone Dataset Sync & Repair

**Sync records from an existing Apify dataset into an existing Pinecone Integrated Embedding index — without needing an OpenAI or Cohere API key.**

VectorRelay is a focused Apify-to-Pinecone sync bridge. It validates source items, maps stable source IDs to deterministic Pinecone record IDs, copies only explicitly selected metadata, retries temporary Pinecone failures, and isolates bad records so one malformed item does not block the rest of the batch.

### What it solves

- Sync existing Apify dataset content to Pinecone without building custom integration glue
- Re-run stable source IDs to update the same deterministic Pinecone records
- Isolate malformed or failed records while valid records continue
- Keep metadata transfer explicit and controlled
- Use Pinecone Integrated Embedding without a separate external embedding API key

### Best fit

- RAG and semantic-search pipelines that already collect or process data in Apify
- Recurring dataset refreshes that need stable Pinecone record IDs
- Re-sync or repair workflows after partial or temporary Pinecone failures
- Teams that want a focused Apify-to-Pinecone integration instead of maintaining custom sync code

### Requirements

- An existing Pinecone index configured for **Integrated Embedding**
- A Pinecone API key with access to that index
- An existing Apify dataset to read
- The Actor input `pineconeTextField` must match the index field mapping (default / commonly `chunk_text`)

### Why VectorRelay

- Deterministic record IDs
- Repeatable update behavior with a stable `sourceIdField`
- Automatic conservative batching
- Temporary Pinecone failure retries
- Record-level failure isolation
- Explicit metadata selection
- No writes back to the source Apify dataset
- Clear sync, failure, and billing reporting
- Each confirmed Pinecone upsert is treated as one `record-synced` pay-per-event unit

### Example source item

```json
{
    "id": "article-123",
    "text": "Example source text",
    "url": "https://example.com/article-123"
}
```

Matching Actor inputs for that item:

- `sourceIdField` = `id`
- `sourceTextField` = `text`

Use a stable `sourceIdField` such as `id` when you want later runs to replace the same Pinecone record.

### Input

| Field | Required | Default | Description |
| --- | --- | --- | --- |
| `pineconeApiKey` | yes | | Apify secret input. Never logged or written to output. |
| `pineconeIndexName` | yes | | Existing Pinecone Integrated Embedding index name. |
| `datasetId` | yes | | Apify dataset to read. Read-only. |
| `sourceTextField` | no | `text` | Dot-notation path to the source text. |
| `pineconeTextField` | no | `chunk_text` | Record field that must match the index embedding field map. |
| `sourceIdField` | no | | Optional primitive identifier path. See [sourceIdField behavior](#sourceidfield-behavior). |
| `metadataFields` | no | `[]` | Explicit metadata fields to copy. |
| `namespace` | no | `__default__` | Pinecone namespace. |
| `maxItems` | no | `1000` | How many source dataset items to **inspect**. Not a maximum successful-sync count. |

### Example Actor input

```json
{
    "pineconeApiKey": "<secret>",
    "pineconeIndexName": "support-articles",
    "datasetId": "abc123dataset",
    "sourceTextField": "text",
    "pineconeTextField": "chunk_text",
    "sourceIdField": "id",
    "metadataFields": ["url"],
    "namespace": "__default__",
    "maxItems": 1000
}
```

### sourceIdField behavior

- If `sourceIdField` is **completely omitted**, VectorRelay generates a deterministic `src_<sha256>` ID from the **whole source item**.
- If `sourceIdField` is **configured**, every processed source item must contain that field with a **non-empty primitive** value (string, number, or boolean).
- If the configured field is missing, empty, an object, an array, or otherwise non-primitive, that item is skipped with `missing_id`.
- There is **no hash fallback** when `sourceIdField` has been configured.

### Repair / update behavior

- For repeatable updates or replacements in Pinecone, configure a **stable** `sourceIdField` such as `id`.
- The same stable source ID maps to the same deterministic Pinecone record ID, so a later run **replaces / upserts** that record.
- If `sourceIdField` is omitted, changing the source item changes the whole-item hash and can create a **new** Pinecone record instead of replacing the old one.
- VectorRelay does **not** delete stale Pinecone records.

### Results / output

The Apify dataset **result count is not** the number of synced Pinecone records.

The output dataset contains:

- one **summary** row
- additional rows only for **skipped**, **failed**, or other **problem** items

Successful source documents are not copied back into the dataset.

| Field | Meaning |
| --- | --- |
| `summary.synced` | Authoritative successful Pinecone sync count |
| `charged` | Successfully synced records billed through the `record-synced` PPE event |
| `unbilledSuccessful` | Successful syncs that were not billed. Should normally be `0` for a successful commercial run |

Example summary:

```json
{
    "type": "summary",
    "status": "completed",
    "datasetId": "abc123dataset",
    "pineconeIndexName": "support-articles",
    "namespace": "__default__",
    "inspected": 1000,
    "valid": 985,
    "synced": 980,
    "skipped": 3,
    "failed": 2,
    "charged": 980,
    "unbilledSuccessful": 0,
    "stoppedByBudget": false,
    "startedAt": "2026-08-16T09:00:00.000Z",
    "finishedAt": "2026-08-16T09:02:00.000Z"
}
```

Example problem record:

```json
{
    "type": "error",
    "sourceIndex": 719,
    "recordId": "src_…",
    "reason": "missing_text",
    "message": "Configured source text field is missing."
}
```

### maxItems

`maxItems` limits how many source dataset items are **inspected**.

It does **not** mean “maximum successful syncs”. Items beyond that inspection limit are not processed during that run.

### Duplicate IDs

If the same source ID appears multiple times in one run, each successful upsert is processed and billed. Pinecone ultimately keeps the last successful upsert for that deterministic record ID.

### Billing model

The Actor is designed for pay-per-event billing based on successfully synced records. Store pricing is configured separately.

The only custom event charged from code is `record-synced`. Charge order is:

1. Check remaining `record-synced` budget
2. Size the next Pinecone batch to that remaining budget (and to 96 records / 1.5 MB)
3. Upsert
4. Charge one `record-synced` per confirmed successful record

Failed, skipped, and budget-stopped records are not charged. A retry of the same in-run upsert is charged once. A later Actor run of the same source item is a new sync operation and may be charged again.

`apify-actor-start` is a platform synthetic event. This Actor never charges it from code.

#### Platform vs local

- **Local / tests:** `Actor.isAtHome()` is false. Dummy PPE and `Infinity` remaining-charge calculations are allowed so development works without Marketplace configuration.
- **Apify platform + `PAY_PER_EVENT`:** before the first Pinecone upsert the Actor fail-closes unless `record-synced` is registered with a finite price greater than 0. Error code: `PPE_EVENT_NOT_CONFIGURED`.
- **Apify platform + FREE:** no extra commercial rule is applied. Charges are no-ops until monetization is enabled.

If `apify-default-dataset-item` remains a paid event, `Actor.pushData()` for the summary or problem records can stop when the charge limit is reached. Removing or unbilling that synthetic event removes this dependency. No extra storage architecture is used in V1.

### Common errors / Troubleshooting

| Code / symptom | What it means | What to check |
| --- | --- | --- |
| `INCOMPATIBLE_INDEX` | The selected Pinecone index is not usable for this Actor. | The index must use **Integrated Embedding**. |
| Field-map mismatch | The index embedding field map does not match the Actor text field. | Pinecone integrated embedding field map must match `pineconeTextField` (default `chunk_text`). |
| Authentication / permission | Pinecone rejected the key or index access. | Check the Pinecone API key and access to the selected index. |
| `PPE_EVENT_NOT_CONFIGURED` | Commercial billing is not ready. | For commercial Apify runs, the `record-synced` PPE event must have a positive configured price. |
| `missing_text` | That source item has no usable text. | The configured `sourceTextField` does not exist or cannot provide usable text. |
| `missing_id` | That source item has no valid ID. | `sourceIdField` was configured, but the item does not contain a valid primitive ID. |

### Data flow / privacy

Selected source text and selected metadata are sent to the user's configured Pinecone index. That is required for synchronization and embedding.

Source data does **not** stay only inside Apify.

The Pinecone API key is an Apify secret input and is never logged or written to output.

### Security

- `pineconeApiKey` is an Apify secret input
- The key is never logged, persisted, or written to the output dataset
- Error messages are redacted before they leave the Actor
- The source dataset is opened read-only
- The Actor does not create or delete Pinecone indexes and does not delete Pinecone records

### Limits

- at most 96 records per Pinecone upsert
- conservative upsert request size 1,500,000 UTF-8 bytes (Pinecone allows 2 MB)
- filterable metadata limited to 32 KB UTF-8 JSON (Pinecone allows 40 KB)
- record IDs are `src_<sha256>` (68 ASCII characters, under the 512-character limit)
- the index `embed.fieldMap` must unambiguously match `pineconeTextField` or the run aborts before writes

### Limitations

- no index creation
- no deletion of stale Pinecone records
- no crawling
- no search/chat
- no external embedding provider

# Actor input Schema

## `pineconeApiKey` (type: `string`):

Pinecone API key. Stored as an Apify secret input. Never logged or written to output.

## `pineconeIndexName` (type: `string`):

Name of an existing Pinecone index configured for integrated embedding. This Actor never creates or deletes indexes.

## `datasetId` (type: `string`):

Apify dataset to read. The dataset is opened read-only.

## `sourceTextField` (type: `string`):

Dot-notation path to the text field in each dataset item. Examples: text, content.text, document.body.

## `pineconeTextField` (type: `string`):

Record field name that must match the index integrated-embedding field mapping (commonly chunk\_text).

## `sourceIdField` (type: `string`):

Optional primitive identifier path (for example id). Omit it entirely to generate a deterministic src\_<sha256> ID from the whole source item. If configured, every processed item must contain that field with a non-empty primitive value; missing, empty, object, array, or other non-primitive values are skipped as missing\_id. There is no hash fallback when this field is configured. Use a stable field so later runs replace the same Pinecone record.

## `metadataFields` (type: `array`):

Explicit dataset fields to copy as Pinecone metadata. Supports string, number, boolean, and string\[]. Nested objects and other arrays are rejected.

## `namespace` (type: `string`):

Target Pinecone namespace. Use **default** for the default namespace.

## `maxItems` (type: `integer`):

Maximum number of source dataset items to inspect in this run. This is an inspection limit, not a maximum number of successful syncs. Items beyond this limit are not processed.

## Actor input object example

```json
{
  "sourceTextField": "text",
  "pineconeTextField": "chunk_text",
  "metadataFields": [],
  "namespace": "__default__",
  "maxItems": 1000
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sourceTextField": "text",
    "pineconeTextField": "chunk_text",
    "namespace": "__default__"
};

// Run the Actor and wait for it to finish
const run = await client.actor("vectorrelay/pinecone-dataset-sync-repair").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "sourceTextField": "text",
    "pineconeTextField": "chunk_text",
    "namespace": "__default__",
}

# Run the Actor and wait for it to finish
run = client.actor("vectorrelay/pinecone-dataset-sync-repair").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sourceTextField": "text",
  "pineconeTextField": "chunk_text",
  "namespace": "__default__"
}' |
apify call vectorrelay/pinecone-dataset-sync-repair --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,vectorrelay/pinecone-dataset-sync-repair"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/D7Ag48c3kwTa3WlpO/builds/oDq2UeUE2bstyTaLX/openapi.json