# Pinecone & Qdrant Vector Database Sync for RAG (`charlesmarkette/vector-db-sync-pinecone-qdrant-rag-loader-delta-updates`) Actor

Load any Apify dataset, like Website Content Crawler pages, into a Pinecone or Qdrant vector database for RAG. Chunks text, embeds with your own OpenAI or Cohere key, and re-embeds only new or changed chunks on later runs, deleting stale ones. $0.18 per 1,000 chunks upserted; unchanged chunks free.

- **URL**: https://apify.com/charlesmarkette/vector-db-sync-pinecone-qdrant-rag-loader-delta-updates.md
- **Developed by:** [Charles Markette](https://apify.com/charlesmarkette) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.18 / 1,000 chunk upserteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

**Vector DB Sync** loads any Apify dataset into a **Pinecone** or **Qdrant** vector database for **RAG** (retrieval-augmented generation), for example the pages [Website Content Crawler](https://apify.com/apify/website-content-crawler) just crawled. It chunks the text, creates **embeddings with your own OpenAI or Cohere key**, and upserts them with the page URL and your metadata. On later runs it uses **delta updates**: only new or changed chunks are embedded and upserted, and stale ones are deleted. It's for developers and AI teams building chatbots, AI assistants, semantic search or agents on top of website, documentation or scraped content.

- **Pinecone** (serverless indexes) and **Qdrant** (Qdrant Cloud or your own server).
- **OpenAI** `text-embedding-3-small` / `-3-large` (with an optional smaller `dimensions`), or **Cohere** `embed-english-v3.0`, `embed-multilingual-v3.0` and the other embed models.
- **Paragraph- and sentence-aware chunking** with overlap. Every chunk keeps the page URL, your metadata fields and its text, ready for LangChain, LlamaIndex or your own code.
- **Delta updates.** Each chunk's fingerprint is remembered, so the next run embeds and upserts **only new or changed chunks**. Unchanged chunks are skipped for free, chunks left over when a page got shorter are deleted, and optionally pages that disappeared from the site are deleted too (with guards against wiping your index by mistake).

Connect it once on your crawler's **Integrations** tab, and your vector index stays in sync with the site.

### Why use Vector DB Sync?

- **Checks before it spends.** Before the first embedding it reads your index (Pinecone `describe_index_stats`) or collection (Qdrant collection info). A wrong API key, a missing index or collection, or a **dimension mismatch** between your model and your index stops the run right away with a message that says what to change. Nothing is embedded, so nothing is wasted.
- **Only pays for changes.** Delta updates skip unchanged chunks before calling the embedding API, so a weekly re-crawl costs a fraction of a full re-sync (see the example below).
- **Retries the right things.** Every request to OpenAI, Cohere, Pinecone and Qdrant is retried on rate limits (HTTP 429) and server errors (5xx) with backoff, honouring the service's `Retry-After`. Errors that retrying can't fix (a bad key, an exhausted OpenAI quota) fail at once, with the fix in the message.
- **Respects every documented limit.** Embedding requests stay within OpenAI's 2,048 inputs and 300,000 tokens per request and 8,192 tokens per input, and within Cohere's 96 texts per call. Pinecone upserts stay within 1,000 vectors and 2 MB per request and 40 KB of metadata per record, and Qdrant writes use `wait=true` and valid UUID point IDs. Over-long chunks are cut safely instead of failing the batch.
- **Crash-safe memory.** The memory of what's in your database is saved after **every** successful upsert batch (and when the platform migrates or aborts the run), so a crash or timeout never makes the next run re-embed what's already stored. If the collection was dropped or the index emptied, the memory is discarded and everything is synced again.
- **Hard to wipe by accident.** A typo in **Text fields** or **ID field** stops the run before anything changes. An item that comes back without text keeps its vectors. Removing items that left the dataset is skipped whenever the run can't be sure which items left, and refuses to remove more than half of the index unless you allow it.
- **One bad item doesn't sink the run.** If the embedding API refuses one chunk, the batch is split to find it, that item is marked `error`, and the rest are synced.
- **Streams the dataset** in pages, so large crawls fit in the default 1 GB of memory.
- **Your keys stay secret.** All API keys are secret inputs, stored encrypted, and masked in every log line, error message and output.

### Use cases

- **Chat with your docs or website:** crawl a documentation site or help center with Website Content Crawler and keep a Pinecone or Qdrant index fresh for your RAG chatbot.
- **Knowledge base for AI agents:** load scraped articles, product pages or support tickets into a vector database your agent can search.
- **Semantic search:** embed any dataset with text fields (reviews, listings, job posts) for similarity search.
- **Scheduled re-indexing:** re-crawl weekly and pay only for pages that changed, with removed pages deleted from the index.

### How to use Vector DB Sync

#### Setup (about 3 minutes)

**1. Your embedding key**

- **OpenAI:** create a key at [platform.openai.com/api-keys](https://platform.openai.com/api-keys) and paste it into **Embedding API key**.
- **Cohere:** set **Embedding provider** to Cohere, create a key at [dashboard.cohere.com/api-keys](https://dashboard.cohere.com/api-keys) and paste it into **Embedding API key**. Use a production key for big datasets: trial keys are rate-limited.

**2a. Pinecone**

1. In the [Pinecone console](https://app.pinecone.io), create a **serverless, dense** index with **dimension** 1536 (for OpenAI `text-embedding-3-small`; 3072 for `-3-large`, 1024 for Cohere `embed-english-v3.0`) and metric **cosine**.
2. Open the index and copy the value under **HOST** (like `docs-example-4zo0ijk.svc.us-east1-aws.pinecone.io`) into **Pinecone index host**.
3. Open **API keys**, create a key and paste it into **Pinecone API key**. Optionally set a **namespace**.

**2b. Qdrant**

1. In the [Qdrant Cloud dashboard](https://cloud.qdrant.io), open your cluster and copy its URL (like `https://xyz-example.eu-central.aws.cloud.qdrant.io:6333`) into **Qdrant URL**. A self-hosted server works too.
2. In the cluster's **API Keys** section, create a key with write access and paste it into **Qdrant API key**. The key is shown only once.
3. Type a **Qdrant collection** name and turn on **Create the collection if missing**: it's created with your model's dimension and Cosine distance.

**3. Test it:** pick a dataset, set **Max items** to 10 and run. Or turn on **Dry run** (free): it reads and chunks the dataset, checks your database connection and dimension, and reports what would be upserted, without embedding or writing anything.

#### Sync after every Website Content Crawler run (integration)

1. Open your Website Content Crawler task, go to the **Integrations** tab and click **Add integration**.
2. Search for **Vector DB Sync**, choose to start it when the run **succeeded**, and set the input. Apify fills in `{{resource.defaultDatasetId}}` with that run's dataset:

```json
{
  "datasetId": "{{resource.defaultDatasetId}}",
  "textFields": ["text"],
  "metadataFields": ["url", "metadata.title"],
  "embeddingProvider": "openai",
  "embeddingModel": "text-embedding-3-small",
  "embeddingApiKey": "sk-…",
  "vectorDb": "pinecone",
  "pineconeApiKey": "pcsk_…",
  "pineconeIndexHost": "docs-example-4zo0ijk.svc.us-east1-aws.pinecone.io",
  "pineconeNamespace": "docs",
  "deltaMode": true,
  "deleteRemoved": true,
  "stateName": "docs-site"
}
```

3. Save, and test the integration on a past crawl.

Every crawl creates a new dataset, but the memory belongs to your **database target** (index host and namespace, or Qdrant URL and collection) plus the **memory name** `stateName`, so each run only embeds what changed since the last one. With `deleteRemoved`, pages that are no longer in the crawl are removed from the index too. That needs a `stateName` (unless this Actor runs as a saved task): give each source its own name, e.g. `docs-site` and `blog`, when several crawls fill one index, so one crawl never deletes the other's pages.

Each vector's metadata then holds `url`, `title`, `text` (the chunk), `item_id`, `chunk_index` and `content_hash`. The vector IDs are stable: `<item_id>#<chunk_index>` in Pinecone, and a UUID derived from the same pair in Qdrant, where `item_id` is a hash of the page URL (or whatever **ID field** you choose).

### Input example

Qdrant with Cohere embeddings and a collection created on the first run:

```json
{
  "datasetId": "aBcDeFgHiJkLmNoPq",
  "textFields": ["text"],
  "metadataFields": ["url", "metadata.title"],
  "chunkSize": 2000,
  "chunkOverlap": 200,
  "embeddingProvider": "cohere",
  "embeddingModel": "embed-multilingual-v3.0",
  "embeddingApiKey": "…",
  "vectorDb": "qdrant",
  "qdrantUrl": "https://xyz-example.eu-central.aws.cloud.qdrant.io:6333",
  "qdrantApiKey": "…",
  "qdrantCollection": "help-center",
  "createCollectionIfMissing": true,
  "stateName": "help-center"
}
```

| Field | Notes |
|---|---|
| Dataset | The dataset to sync. In integrations, `{{resource.defaultDatasetId}}`. |
| Text fields | Fields to embed, joined with a blank line (default `["text"]`). Dots for nested fields. |
| Metadata fields | Stored with every chunk, under the field's last part (`metadata.title` → `title`; when two fields share a last part, the whole path with `_`). `null` and empty values are left out; objects and mixed lists are stored as JSON text. The item's `url` is always stored when it has one. |
| ID field | Identifies an item across runs (default `url`). `#text` = identify items by their text (a changed text is then a new item; the old one goes only with **Delete items no longer in the dataset**). Items without a value are reported as `error`; a second item with the same ID is reported as `duplicate` and skipped. |
| Chunk size / overlap | Default 2,000 characters with 200 overlap. Size 100 to 8,000; overlap at most half the size. Splits at paragraphs, then lines, sentences and words. |
| Embedding provider / model / API key / dimensions | `openai` (default model `text-embedding-3-small`) or `cohere` (default `embed-english-v3.0`, `input_type` `search_document`). **Dimensions** works with OpenAI `text-embedding-3` models and Cohere `embed-v4.0`. |
| Vector database | `pinecone` or `qdrant`. |
| Pinecone API key / index host / namespace | The index host from the console (not the index name). Empty namespace = the default namespace. |
| Qdrant URL / API key / collection / vector name / create if missing | **Vector name** only for collections with named vectors. |
| Delta updates | Default on. Off = re-embed and upsert every chunk once (the memory is still updated). |
| Delete items no longer in the dataset | Default off. Deletes the vectors of items that earlier runs with the same memory synced and that are missing now. Needs a **Memory name** (or a saved task). Nothing is deleted when the run didn't read the whole dataset (**Max items** smaller than the dataset, or a cost stop), when the dataset is empty, when any item had no ID (an item with an ID but no text keeps its vectors and doesn't block this), or when more than half of the remembered items would go. The counts (in the memory, in the dataset, missing) are always logged. |
| Allow removing more than half | Default off. Lets the option above remove more than 50% of the remembered items in one run, for a site that really shrank. |
| Dry run | Counts what would be upserted and deleted. No embedding calls, no writes, free. |
| Max items | Sync only the first N items. |
| Store the chunk text | Default on: the chunk is stored under `text` in the metadata. |
| Memory name | Required for **Delete items no longer in the dataset** (unless the Actor runs as a saved task), optional otherwise. The memory always belongs to the database target, plus this name, else the saved task. Without either, all runs into that target share one memory. Use one name per source. |

All API keys are secret inputs, stored encrypted.

### Output example

The run's **dataset** has one row per dataset item:

```json
[
  {
    "id": "https://docs.example.com/getting-started",
    "url": "https://docs.example.com/getting-started",
    "itemId": "5d41402abc4b2a76b9719d911017c592",
    "status": "upserted",
    "chunks": 4,
    "chunksUpserted": 1,
    "chunksUnchanged": 3,
    "chunksDeleted": 0,
    "message": null
  },
  {
    "id": "https://docs.example.com/pricing",
    "url": "https://docs.example.com/pricing",
    "itemId": "7d793037a0760186574b0282f2f435e7",
    "status": "unchanged",
    "chunks": 2,
    "chunksUpserted": 0,
    "chunksUnchanged": 2,
    "chunksDeleted": 0,
    "message": null
  }
]
```

- `status` is `upserted`, `unchanged`, `updated` (only old chunks deleted), `skipped` (no text; its vectors are kept), `duplicate`, `error`, `stopped` (cost limit), or `would upsert` / `would update` in a dry run. `itemId` is the stable hash used in the vector IDs.
- The key-value store **`OUTPUT`** holds items read and per status, chunks total, upserted, unchanged and deleted (`chunksDeleted` for chunks left over by shorter items, `chunksDeletedRemoved` for items no longer in the dataset), embedding tokens (as reported by OpenAI or Cohere), requests and retries, the target, the dimension, and timings per stage (`read`, `chunk`, `embed`, `upsert`, `delete`, `state`, `total`). It's also written when a run fails partway.
- Without any dataset or keys, the run returns a single row with setup instructions and does nothing else.

### How much does it cost to load data into Pinecone or Qdrant?

Vector DB Sync uses pay-per-event pricing: **$0.00018 per chunk upserted** ($0.18 per 1,000), plus Apify's $0.00005 run start fee. You pay only for chunks that were actually embedded and written to your database. Unchanged chunks, deletions, dry runs, chunks that failed to embed or upsert, and the result rows in the dataset are free (an item marked `error` is charged only for the chunks of it that were stored). Embedding costs are billed by OpenAI or Cohere to your own key.

**Worked example:** you crawl 1,000 documentation pages (about 3,000 chunks) every week, and 30 pages change each week.

*Examples use the Free-plan rate. Where an example doesn't add it, each run also has its run-start charge, counted per GB of Actor memory (minimum one; one at the default memory); Apify Proxy (where used) is billed separately.*

| | Chunks embedded and upserted | Vector DB Sync charge |
|---|---|---|
| First run | 3,000 | 3,000 × $0.00018 = $0.54 |
| Each weekly run after that (delta mode) | about 90, plus deleting the chunks of pages that got shorter | about $0.016 |
| Each weekly run without delta updates | 3,000 | $0.54 |

The other ~2,900 chunks are compared by fingerprint and skipped: no embedding call, no upsert, no charge, and nothing on your OpenAI or Cohere bill.

If you set a maximum cost per run, the Actor checks it **before** each embedding request and embeds only as many chunks as it can still bill, so you never pay your embedding provider for chunks that weren't stored. The run then stops cleanly, the stopped items say so, and the next run continues where it stopped.

### FAQ

#### The run stopped with a dimension mismatch.

Your index or collection was created with a different vector size than the model produces. Create the Pinecone index with the model's dimension (1536 for `text-embedding-3-small`, 3072 for `-3-large`, 1024 for Cohere `embed-english-v3.0`), or set **Dimensions** to match your index (OpenAI `text-embedding-3` models and Cohere `embed-v4.0`).

#### "No Pinecone index at … (HTTP 404)".

Paste the index **host** shown on the index's page in the Pinecone console (like `my-index-abc1234.svc….pinecone.io`), not the index name.

#### Everything was re-embedded, although the site barely changed.

A chunk is re-embedded when its text, its metadata fields or the embedding settings change. Don't pick metadata fields that change on every crawl, like timestamps. Changing the chunk size also re-embeds everything.

#### "Delete items no longer in the dataset" didn't delete anything.

It needs a **Memory name** (or a saved task), and for safety it deletes nothing when the run didn't read the whole dataset, when the dataset is empty, when any item had no ID, or when more than half of the remembered items would go. The run log shows the counts; turn on **Allow removing more than half** if the site really shrank.

#### My OpenAI quota or Cohere trial limit was hit.

An exhausted OpenAI quota fails the run at once. Cohere trial keys are rate-limited; use a production key for big datasets. Thanks to the memory, the next run skips everything already synced.

#### Can I use it with LangChain or LlamaIndex?

Yes. The chunk text is stored under `text` in the metadata (what LangChain and most RAG code expect), next to `url` and your metadata fields.

#### Does it work with other vector databases?

Pinecone and Qdrant only.

### Limitations

- **Memory:** about 200 bytes per synced item, so the default 1 GB handles roughly 2 million items per memory. Items with more than 10,000 chunks (about 20 MB of text) are cut at 10,000 chunks.
- **What counts as a change:** a chunk is re-embedded when its text, its metadata fields, the embedding settings (provider, model, dimensions) or **Store the chunk text** change (switching the database type changes the metadata size limit, which counts too). Don't pick metadata fields that change on every crawl, like timestamps. Changing the chunk size re-embeds everything and deletes the old chunks. Inserting text near the top of a page shifts its chunks, so the page's later chunks are re-embedded too.
- **The memory trusts itself, mostly.** It is discarded automatically (with a warning) when the Qdrant collection is missing or has 0 points, or the Pinecone index reports 0 vectors. The Pinecone check uses the whole index's `totalVectorCount` from `describe_index_stats` (not the namespace's count), and Pinecone's counts can lag a little behind writes, so a run started seconds after a first sync may re-embed. If you delete only some vectors by hand, turn **Delta updates** off for one run (or change the memory name) so everything is upserted again. Don't run two syncs with the same memory at the same time.
- **OpenAI:** up to 2,048 inputs and 300,000 tokens per request, 8,192 tokens per input. Without a tokenizer, the Actor treats every UTF-8 byte as a possible token, so a chunk is cut only if it's over 8,191 bytes (never at the default chunk size). An exhausted quota or spend limit fails the run at once.
- **Cohere:** up to 96 texts per request. The v3 models read at most 512 tokens per chunk (about 2,000 characters of English) and cut the rest (`truncate: END`), so keep **Chunk size** at 2,000 or less with them. `Dimensions` needs `embed-v4.0`.
- **Pinecone:** dense serverless indexes, API version 2026-07; up to 1,000 vectors and 2 MB per upsert, 1,000 IDs per delete, and 40 KB of metadata per vector. Metadata over the limit is shortened (longest values first; the bookkeeping fields are never cut) and counted in `chunksMetadataShortened`. Metadata values are strings, numbers, booleans or lists of strings, and names never start with `$`, as Pinecone requires. A delete answered with 404 (namespace already gone) counts as done.
- **Qdrant:** single unnamed vector or one named vector; collections created by the Actor use Cosine distance. Writes wait until they're applied (`wait=true`). Up to 256 points per request.
- **Retries:** up to 7 attempts per request on HTTP 429, 5xx and network errors, waiting at most 90 seconds at a time and 5 minutes in total. A service that asks for a longer wait fails the run, and the next run skips everything already synced.
- **URLs with a user name and password** (`https://user:pass@…`) work for Qdrant, but the credentials are removed from logs, OUTPUT and the memory key.
- After the first page, the dataset is read with only the top-level fields the sync uses (text, metadata, ID and `url`).
- **Restarts:** if the platform restarts a run, rows already in the output dataset aren't added twice, and chunks already upserted are skipped by the memory. Charges carry an idempotency key per upsert batch.
- Not affiliated with Pinecone, Qdrant, OpenAI or Cohere.

### Related Actors

Vector DB Sync is part of a small suite of Actors that work together after any scraper:

| Actor | Use it to |
|---|---|
| [Merge & Deduplicate Datasets](https://apify.com/charlesmarkette/merge-deduplicate-datasets-only-new-items) | Merge several crawls and remove duplicate pages before loading them. |
| [Send Email Report & Slack, Discord, Teams, Telegram Alerts](https://apify.com/charlesmarkette/send-email-report-slack-discord-teams-telegram-alerts) | Get a Slack or email message after each sync. |
| [Google Sheets Sync](https://apify.com/charlesmarkette/google-sheets-sync) | Keep a reviewable list of crawled pages in a Google Sheet. |
| [Dataset Delivery](https://apify.com/charlesmarkette/dataset-delivery-export-to-sftp-ftp-dropbox-google-drive) | Archive each crawl as a JSONL file on SFTP, Dropbox or Google Drive. |
| [SEC Funding Rounds Tracker](https://apify.com/charlesmarkette/sec-form-d-funding-tracker) | Track newly funded companies from SEC Form D, Form C and Reg A filings. |
| [PDF & Image to Text](https://apify.com/charlesmarkette/pdf-image-to-text-markdown-ocr) | Extract text or Markdown from PDFs and scanned documents (with OCR) before loading them for RAG. |

# Actor input Schema

## `datasetId` (type: `string`):

The dataset to sync, e.g. the output of Website Content Crawler. In an integration (Integrations → Connect Actor or task), use {{resource.defaultDatasetId}}.

## `textFields` (type: `array`):

Fields to embed, joined with a blank line in this order. Dots for nested fields ('metadata.description'). Website Content Crawler puts the page text in 'text' (or 'markdown').

## `metadataFields` (type: `array`):

Fields stored with every chunk for filtering and citations. Dots for nested fields; each is stored under its last part ('metadata.title' → 'title'). The item's 'url' is always stored when it has one. Don't pick fields that change on every crawl (like timestamps): a changed field re-embeds the item.

## `idField` (type: `string`):

The field that identifies an item across runs (default 'url'). It's hashed into stable vector IDs, so the same page always overwrites its own vectors. Enter #text to identify items by their text instead (then a changed text counts as a new item, and the old one is only removed with 'Delete items no longer in the dataset').

## `chunkSize` (type: `integer`):

Maximum characters per chunk. 2,000 is about 500 tokens of English.

## `chunkOverlap` (type: `integer`):

Characters repeated from the end of one chunk at the start of the next (whole sentences or words where possible). At most half the chunk size.

## `embeddingProvider` (type: `string`):

Who turns your chunks into vectors.

## `embeddingModel` (type: `string`):

Empty = text-embedding-3-small (OpenAI) or embed-english-v3.0 (Cohere). OpenAI: text-embedding-3-small (1536 dimensions), text-embedding-3-large (3072). Cohere: embed-english-v3.0 or embed-multilingual-v3.0 (1024), the -light-v3.0 versions (384), embed-v4.0 (1536).

## `embeddingApiKey` (type: `string`):

Your OpenAI (sk-…) or Cohere API key. Stored encrypted, never logged.

## `dimensions` (type: `integer`):

Optional smaller vector size: OpenAI text-embedding-3 models (any size up to the model's), Cohere embed-v4.0 (256, 512, 1024 or 1536). Must match your index. Empty = the model's own size.

## `vectorDb` (type: `string`):

Where the vectors go. Fill in the fields for that database below.

## `pineconeApiKey` (type: `string`):

Pinecone console → API keys. Stored encrypted, never logged.

## `pineconeIndexHost` (type: `string`):

The HOST shown on your index's page in the Pinecone console, like my-index-abc1234.svc.aped-4627-b74a.pinecone.io. The index must be dense, with your model's dimension.

## `pineconeNamespace` (type: `string`):

Optional. Empty = the default namespace.

## `qdrantUrl` (type: `string`):

Your cluster URL, like https://xyz-example.eu-central.aws.cloud.qdrant.io:6333 (Qdrant Cloud) or https://qdrant.example.com:6333.

## `qdrantApiKey` (type: `string`):

Your Qdrant API key (empty for a server without authentication). Stored encrypted, never logged.

## `qdrantCollection` (type: `string`):

The collection to fill.

## `createCollectionIfMissing` (type: `boolean`):

Creates the Qdrant collection with your model's dimension and Cosine distance when it doesn't exist.

## `qdrantVectorName` (type: `string`):

Only for collections with named vectors: the vector to fill. Empty = the collection's single unnamed vector.

## `deltaMode` (type: `boolean`):

Remembers a fingerprint of every chunk in your database, and embeds and upserts only new or changed chunks. Unchanged chunks are skipped for free. Turn off to re-embed everything once.

## `deleteRemoved` (type: `boolean`):

Deletes the vectors of items that earlier runs with the same 'Memory name' synced and that are missing from this dataset, e.g. pages removed from the site. Needs a 'Memory name' (unless the Actor runs as a saved task). For safety it deletes nothing when the run didn't read the whole dataset (Max items smaller than the dataset, or a cost stop), when the dataset is empty, when any item had no ID (an item with an ID but no text keeps its vectors and does not block this), or when more than half of the remembered items would go (see the next option).

## `allowLargeRemoval` (type: `boolean`):

Lets 'Delete items no longer in the dataset' remove more than 50% of the remembered items in one run. Leave off unless the site really shrank: a half-empty dataset (a failed crawl) would otherwise wipe your index.

## `dryRun` (type: `boolean`):

Reads and chunks the dataset and reports what would be upserted and deleted, without calling the embedding API or writing to the database (free). Checks the database connection and dimension when it's filled in.

## `maxItems` (type: `integer`):

Optional: sync only the first N dataset items (handy for a first test).

## `storeText` (type: `boolean`):

Stores each chunk's text in the metadata under 'text' (what LangChain and most RAG code expect). Turn off to store only the vectors and your metadata fields. Changing this re-upserts every chunk.

## `stateName` (type: `string`):

Optional, but required for 'Delete items no longer in the dataset' (unless the Actor runs as a saved task). Runs with the same name and the same database target share one memory of synced chunks; use one name per source (e.g. 'docs-site') when several sources fill one index. Empty = the memory belongs to the saved task, or else to the database target alone.

## Actor input object example

```json
{
  "textFields": [
    "text"
  ],
  "metadataFields": [
    "url",
    "metadata.title"
  ],
  "idField": "url",
  "chunkSize": 2000,
  "chunkOverlap": 200,
  "embeddingProvider": "openai",
  "vectorDb": "pinecone",
  "createCollectionIfMissing": false,
  "deltaMode": true,
  "deleteRemoved": false,
  "allowLargeRemoval": false,
  "dryRun": false,
  "storeText": true
}
```

# Actor output Schema

## `items` (type: `string`):

Status and chunk counts for each dataset item.

## `summary` (type: `string`):

Items read, chunks total, upserted, unchanged and deleted, embedding tokens, and timings per stage.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "textFields": [
        "text"
    ],
    "metadataFields": [
        "url",
        "metadata.title"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("charlesmarkette/vector-db-sync-pinecone-qdrant-rag-loader-delta-updates").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "textFields": ["text"],
    "metadataFields": [
        "url",
        "metadata.title",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("charlesmarkette/vector-db-sync-pinecone-qdrant-rag-loader-delta-updates").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "textFields": [
    "text"
  ],
  "metadataFields": [
    "url",
    "metadata.title"
  ]
}' |
apify call charlesmarkette/vector-db-sync-pinecone-qdrant-rag-loader-delta-updates --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,charlesmarkette/vector-db-sync-pinecone-qdrant-rag-loader-delta-updates"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/2R00msbWg3E9ArLUx/builds/wndtfo6FTl3wFDui5/openapi.json