VectorRelay – Pinecone Dataset Sync & Repair

Sync records from an existing Apify dataset into an existing Pinecone Integrated Embedding index — without needing an OpenAI or Cohere API key.

VectorRelay is a focused Apify-to-Pinecone sync bridge. It validates source items, maps stable source IDs to deterministic Pinecone record IDs, copies only explicitly selected metadata, retries temporary Pinecone failures, and isolates bad records so one malformed item does not block the rest of the batch.

What it solves

• Sync existing Apify dataset content to Pinecone without building custom integration glue
• Re-run stable source IDs to update the same deterministic Pinecone records
• Isolate malformed or failed records while valid records continue
• Keep metadata transfer explicit and controlled
• Use Pinecone Integrated Embedding without a separate external embedding API key

Best fit

• RAG and semantic-search pipelines that already collect or process data in Apify
• Recurring dataset refreshes that need stable Pinecone record IDs
• Re-sync or repair workflows after partial or temporary Pinecone failures
• Teams that want a focused Apify-to-Pinecone integration instead of maintaining custom sync code

Requirements

• An existing Pinecone index configured for Integrated Embedding
• A Pinecone API key with access to that index
• An existing Apify dataset to read
• The Actor input pineconeTextField must match the index field mapping (default / commonly chunk_text)

Why VectorRelay

• Deterministic record IDs
• Repeatable update behavior with a stable sourceIdField
• Automatic conservative batching
• Temporary Pinecone failure retries
• Record-level failure isolation
• Explicit metadata selection
• No writes back to the source Apify dataset
• Clear sync, failure, and billing reporting
• Each confirmed Pinecone upsert is treated as one record-synced pay-per-event unit

Example source item

{
    "id": "article-123",
    "text": "Example source text",
    "url": "https://example.com/article-123"
}

Matching Actor inputs for that item:

• sourceIdField = id
• sourceTextField = text

Use a stable sourceIdField such as id when you want later runs to replace the same Pinecone record.

Input

Field	Required	Default	Description
pineconeApiKey	yes		Apify secret input. Never logged or written to output.
pineconeIndexName	yes		Existing Pinecone Integrated Embedding index name.
datasetId	yes		Apify dataset to read. Read-only.
sourceTextField	no	text	Dot-notation path to the source text.
pineconeTextField	no	chunk_text	Record field that must match the index embedding field map.
sourceIdField	no		Optional primitive identifier path. See sourceIdField behavior.
metadataFields	no	[]	Explicit metadata fields to copy.
namespace	no	__default__	Pinecone namespace.
maxItems	no	1000	How many source dataset items to inspect. Not a maximum successful-sync count.

Example Actor input

{
    "pineconeApiKey": "<secret>",
    "pineconeIndexName": "support-articles",
    "datasetId": "abc123dataset",
    "sourceTextField": "text",
    "pineconeTextField": "chunk_text",
    "sourceIdField": "id",
    "metadataFields": ["url"],
    "namespace": "__default__",
    "maxItems": 1000
}

sourceIdField behavior

• If sourceIdField is completely omitted, VectorRelay generates a deterministic src_<sha256> ID from the whole source item.
• If sourceIdField is configured, every processed source item must contain that field with a non-empty primitive value (string, number, or boolean).
• If the configured field is missing, empty, an object, an array, or otherwise non-primitive, that item is skipped with missing_id.
• There is no hash fallback when sourceIdField has been configured.

Repair / update behavior

• For repeatable updates or replacements in Pinecone, configure a stable sourceIdField such as id.
• The same stable source ID maps to the same deterministic Pinecone record ID, so a later run replaces / upserts that record.
• If sourceIdField is omitted, changing the source item changes the whole-item hash and can create a new Pinecone record instead of replacing the old one.
• VectorRelay does not delete stale Pinecone records.

Results / output

The Apify dataset result count is not the number of synced Pinecone records.

The output dataset contains:

• one summary row
• additional rows only for skipped, failed, or other problem items

Successful source documents are not copied back into the dataset.

Field	Meaning
summary.synced	Authoritative successful Pinecone sync count
charged	Successfully synced records billed through the record-synced PPE event
unbilledSuccessful	Successful syncs that were not billed. Should normally be 0 for a successful commercial run

Example summary:

{
    "type": "summary",
    "status": "completed",
    "datasetId": "abc123dataset",
    "pineconeIndexName": "support-articles",
    "namespace": "__default__",
    "inspected": 1000,
    "valid": 985,
    "synced": 980,
    "skipped": 3,
    "failed": 2,
    "charged": 980,
    "unbilledSuccessful": 0,
    "stoppedByBudget": false,
    "startedAt": "2026-08-16T09:00:00.000Z",
    "finishedAt": "2026-08-16T09:02:00.000Z"
}

Example problem record:

{
    "type": "error",
    "sourceIndex": 719,
    "recordId": "src_…",
    "reason": "missing_text",
    "message": "Configured source text field is missing."
}

maxItems

maxItems limits how many source dataset items are inspected.

It does not mean “maximum successful syncs”. Items beyond that inspection limit are not processed during that run.

Duplicate IDs

If the same source ID appears multiple times in one run, each successful upsert is processed and billed. Pinecone ultimately keeps the last successful upsert for that deterministic record ID.

Billing model

The Actor is designed for pay-per-event billing based on successfully synced records. Store pricing is configured separately.

The only custom event charged from code is record-synced. Charge order is:

1. Check remaining record-synced budget
2. Size the next Pinecone batch to that remaining budget (and to 96 records / 1.5 MB)
3. Upsert
4. Charge one record-synced per confirmed successful record

Failed, skipped, and budget-stopped records are not charged. A retry of the same in-run upsert is charged once. A later Actor run of the same source item is a new sync operation and may be charged again.

apify-actor-start is a platform synthetic event. This Actor never charges it from code.

Platform vs local

• Local / tests: Actor.isAtHome() is false. Dummy PPE and Infinity remaining-charge calculations are allowed so development works without Marketplace configuration.
• Apify platform + PAY_PER_EVENT: before the first Pinecone upsert the Actor fail-closes unless record-synced is registered with a finite price greater than 0. Error code: PPE_EVENT_NOT_CONFIGURED.
• Apify platform + FREE: no extra commercial rule is applied. Charges are no-ops until monetization is enabled.

If apify-default-dataset-item remains a paid event, Actor.pushData() for the summary or problem records can stop when the charge limit is reached. Removing or unbilling that synthetic event removes this dependency. No extra storage architecture is used in V1.

Common errors / Troubleshooting

Code / symptom	What it means	What to check
INCOMPATIBLE_INDEX	The selected Pinecone index is not usable for this Actor.	The index must use Integrated Embedding.
Field-map mismatch	The index embedding field map does not match the Actor text field.	Pinecone integrated embedding field map must match pineconeTextField (default chunk_text).
Authentication / permission	Pinecone rejected the key or index access.	Check the Pinecone API key and access to the selected index.
PPE_EVENT_NOT_CONFIGURED	Commercial billing is not ready.	For commercial Apify runs, the record-synced PPE event must have a positive configured price.
missing_text	That source item has no usable text.	The configured sourceTextField does not exist or cannot provide usable text.
missing_id	That source item has no valid ID.	sourceIdField was configured, but the item does not contain a valid primitive ID.

Data flow / privacy

Selected source text and selected metadata are sent to the user's configured Pinecone index. That is required for synchronization and embedding.

Source data does not stay only inside Apify.

The Pinecone API key is an Apify secret input and is never logged or written to output.

Security

• pineconeApiKey is an Apify secret input
• The key is never logged, persisted, or written to the output dataset
• Error messages are redacted before they leave the Actor
• The source dataset is opened read-only
• The Actor does not create or delete Pinecone indexes and does not delete Pinecone records

Limits

• at most 96 records per Pinecone upsert
• conservative upsert request size 1,500,000 UTF-8 bytes (Pinecone allows 2 MB)
• filterable metadata limited to 32 KB UTF-8 JSON (Pinecone allows 40 KB)
• record IDs are src_<sha256> (68 ASCII characters, under the 512-character limit)
• the index embed.fieldMap must unambiguously match pineconeTextField or the run aborts before writes

Limitations

• no index creation
• no deletion of stale Pinecone records
• no crawling
• no search/chat
• no external embedding provider