Pinecone & Qdrant Vector Database Sync for RAG

Vector DB Sync loads any Apify dataset into a Pinecone or Qdrant vector database for RAG (retrieval-augmented generation), for example the pages Website Content Crawler just crawled. It chunks the text, creates embeddings with your own OpenAI or Cohere key, and upserts them with the page URL and your metadata. On later runs it uses delta updates: only new or changed chunks are embedded and upserted, and stale ones are deleted. It's for developers and AI teams building chatbots, AI assistants, semantic search or agents on top of website, documentation or scraped content.

• Pinecone (serverless indexes) and Qdrant (Qdrant Cloud or your own server).
• OpenAI text-embedding-3-small / -3-large (with an optional smaller dimensions), or Cohere embed-english-v3.0, embed-multilingual-v3.0 and the other embed models.
• Paragraph- and sentence-aware chunking with overlap. Every chunk keeps the page URL, your metadata fields and its text, ready for LangChain, LlamaIndex or your own code.
• Delta updates. Each chunk's fingerprint is remembered, so the next run embeds and upserts only new or changed chunks. Unchanged chunks are skipped for free, chunks left over when a page got shorter are deleted, and optionally pages that disappeared from the site are deleted too (with guards against wiping your index by mistake).

Connect it once on your crawler's Integrations tab, and your vector index stays in sync with the site.

Why use Vector DB Sync?

• Checks before it spends. Before the first embedding it reads your index (Pinecone describe_index_stats) or collection (Qdrant collection info). A wrong API key, a missing index or collection, or a dimension mismatch between your model and your index stops the run right away with a message that says what to change. Nothing is embedded, so nothing is wasted.
• Only pays for changes. Delta updates skip unchanged chunks before calling the embedding API, so a weekly re-crawl costs a fraction of a full re-sync (see the example below).
• Retries the right things. Every request to OpenAI, Cohere, Pinecone and Qdrant is retried on rate limits (HTTP 429) and server errors (5xx) with backoff, honouring the service's Retry-After. Errors that retrying can't fix (a bad key, an exhausted OpenAI quota) fail at once, with the fix in the message.
• Respects every documented limit. Embedding requests stay within OpenAI's 2,048 inputs and 300,000 tokens per request and 8,192 tokens per input, and within Cohere's 96 texts per call. Pinecone upserts stay within 1,000 vectors and 2 MB per request and 40 KB of metadata per record, and Qdrant writes use wait=true and valid UUID point IDs. Over-long chunks are cut safely instead of failing the batch.
• Crash-safe memory. The memory of what's in your database is saved after every successful upsert batch (and when the platform migrates or aborts the run), so a crash or timeout never makes the next run re-embed what's already stored. If the collection was dropped or the index emptied, the memory is discarded and everything is synced again.
• Hard to wipe by accident. A typo in Text fields or ID field stops the run before anything changes. An item that comes back without text keeps its vectors. Removing items that left the dataset is skipped whenever the run can't be sure which items left, and refuses to remove more than half of the index unless you allow it.
• One bad item doesn't sink the run. If the embedding API refuses one chunk, the batch is split to find it, that item is marked error, and the rest are synced.
• Streams the dataset in pages, so large crawls fit in the default 1 GB of memory.
• Your keys stay secret. All API keys are secret inputs, stored encrypted, and masked in every log line, error message and output.

Use cases

• Chat with your docs or website: crawl a documentation site or help center with Website Content Crawler and keep a Pinecone or Qdrant index fresh for your RAG chatbot.
• Knowledge base for AI agents: load scraped articles, product pages or support tickets into a vector database your agent can search.
• Semantic search: embed any dataset with text fields (reviews, listings, job posts) for similarity search.
• Scheduled re-indexing: re-crawl weekly and pay only for pages that changed, with removed pages deleted from the index.

How to use Vector DB Sync

Setup (about 3 minutes)

1. Your embedding key

• OpenAI: create a key at platform.openai.com/api-keys and paste it into Embedding API key.
• Cohere: set Embedding provider to Cohere, create a key at dashboard.cohere.com/api-keys and paste it into Embedding API key. Use a production key for big datasets: trial keys are rate-limited.

2a. Pinecone

1. In the Pinecone console, create a serverless, dense index with dimension 1536 (for OpenAI text-embedding-3-small; 3072 for -3-large, 1024 for Cohere embed-english-v3.0) and metric cosine.
2. Open the index and copy the value under HOST (like docs-example-4zo0ijk.svc.us-east1-aws.pinecone.io) into Pinecone index host.
3. Open API keys, create a key and paste it into Pinecone API key. Optionally set a namespace.

2b. Qdrant

1. In the Qdrant Cloud dashboard, open your cluster and copy its URL (like https://xyz-example.eu-central.aws.cloud.qdrant.io:6333) into Qdrant URL. A self-hosted server works too.
2. In the cluster's API Keys section, create a key with write access and paste it into Qdrant API key. The key is shown only once.
3. Type a Qdrant collection name and turn on Create the collection if missing: it's created with your model's dimension and Cosine distance.

3. Test it: pick a dataset, set Max items to 10 and run. Or turn on Dry run (free): it reads and chunks the dataset, checks your database connection and dimension, and reports what would be upserted, without embedding or writing anything.

Sync after every Website Content Crawler run (integration)

1. Open your Website Content Crawler task, go to the Integrations tab and click Add integration.
2. Search for Vector DB Sync, choose to start it when the run succeeded, and set the input. Apify fills in {{resource.defaultDatasetId}} with that run's dataset:

{
  "datasetId": "{{resource.defaultDatasetId}}",
  "textFields": ["text"],
  "metadataFields": ["url", "metadata.title"],
  "embeddingProvider": "openai",
  "embeddingModel": "text-embedding-3-small",
  "embeddingApiKey": "sk-…",
  "vectorDb": "pinecone",
  "pineconeApiKey": "pcsk_…",
  "pineconeIndexHost": "docs-example-4zo0ijk.svc.us-east1-aws.pinecone.io",
  "pineconeNamespace": "docs",
  "deltaMode": true,
  "deleteRemoved": true,
  "stateName": "docs-site"
}

3. Save, and test the integration on a past crawl.

Every crawl creates a new dataset, but the memory belongs to your database target (index host and namespace, or Qdrant URL and collection) plus the memory name stateName, so each run only embeds what changed since the last one. With deleteRemoved, pages that are no longer in the crawl are removed from the index too. That needs a stateName (unless this Actor runs as a saved task): give each source its own name, e.g. docs-site and blog, when several crawls fill one index, so one crawl never deletes the other's pages.

Each vector's metadata then holds url, title, text (the chunk), item_id, chunk_index and content_hash. The vector IDs are stable: <item_id>#<chunk_index> in Pinecone, and a UUID derived from the same pair in Qdrant, where item_id is a hash of the page URL (or whatever ID field you choose).

Input example

Qdrant with Cohere embeddings and a collection created on the first run:

{
  "datasetId": "aBcDeFgHiJkLmNoPq",
  "textFields": ["text"],
  "metadataFields": ["url", "metadata.title"],
  "chunkSize": 2000,
  "chunkOverlap": 200,
  "embeddingProvider": "cohere",
  "embeddingModel": "embed-multilingual-v3.0",
  "embeddingApiKey": "…",
  "vectorDb": "qdrant",
  "qdrantUrl": "https://xyz-example.eu-central.aws.cloud.qdrant.io:6333",
  "qdrantApiKey": "…",
  "qdrantCollection": "help-center",
  "createCollectionIfMissing": true,
  "stateName": "help-center"
}

Field	Notes
Dataset	The dataset to sync. In integrations, {{resource.defaultDatasetId}}.
Text fields	Fields to embed, joined with a blank line (default ["text"]). Dots for nested fields.
Metadata fields	Stored with every chunk, under the field's last part (metadata.title → title; when two fields share a last part, the whole path with _). null and empty values are left out; objects and mixed lists are stored as JSON text. The item's url is always stored when it has one.
ID field	Identifies an item across runs (default url). #text = identify items by their text (a changed text is then a new item; the old one goes only with Delete items no longer in the dataset). Items without a value are reported as error; a second item with the same ID is reported as duplicate and skipped.
Chunk size / overlap	Default 2,000 characters with 200 overlap. Size 100 to 8,000; overlap at most half the size. Splits at paragraphs, then lines, sentences and words.
Embedding provider / model / API key / dimensions	openai (default model text-embedding-3-small) or cohere (default embed-english-v3.0, input_type search_document). Dimensions works with OpenAI text-embedding-3 models and Cohere embed-v4.0.
Vector database	pinecone or qdrant.
Pinecone API key / index host / namespace	The index host from the console (not the index name). Empty namespace = the default namespace.
Qdrant URL / API key / collection / vector name / create if missing	Vector name only for collections with named vectors.
Delta updates	Default on. Off = re-embed and upsert every chunk once (the memory is still updated).
Delete items no longer in the dataset	Default off. Deletes the vectors of items that earlier runs with the same memory synced and that are missing now. Needs a Memory name (or a saved task). Nothing is deleted when the run didn't read the whole dataset (Max items smaller than the dataset, or a cost stop), when the dataset is empty, when any item had no ID (an item with an ID but no text keeps its vectors and doesn't block this), or when more than half of the remembered items would go. The counts (in the memory, in the dataset, missing) are always logged.
Allow removing more than half	Default off. Lets the option above remove more than 50% of the remembered items in one run, for a site that really shrank.
Dry run	Counts what would be upserted and deleted. No embedding calls, no writes, free.
Max items	Sync only the first N items.
Store the chunk text	Default on: the chunk is stored under text in the metadata.
Memory name	Required for Delete items no longer in the dataset (unless the Actor runs as a saved task), optional otherwise. The memory always belongs to the database target, plus this name, else the saved task. Without either, all runs into that target share one memory. Use one name per source.

All API keys are secret inputs, stored encrypted.

Output example

The run's dataset has one row per dataset item:

[
  {
    "id": "https://docs.example.com/getting-started",
    "url": "https://docs.example.com/getting-started",
    "itemId": "5d41402abc4b2a76b9719d911017c592",
    "status": "upserted",
    "chunks": 4,
    "chunksUpserted": 1,
    "chunksUnchanged": 3,
    "chunksDeleted": 0,
    "message": null
  },
  {
    "id": "https://docs.example.com/pricing",
    "url": "https://docs.example.com/pricing",
    "itemId": "7d793037a0760186574b0282f2f435e7",
    "status": "unchanged",
    "chunks": 2,
    "chunksUpserted": 0,
    "chunksUnchanged": 2,
    "chunksDeleted": 0,
    "message": null
  }
]

• status is upserted, unchanged, updated (only old chunks deleted), skipped (no text; its vectors are kept), duplicate, error, stopped (cost limit), or would upsert / would update in a dry run. itemId is the stable hash used in the vector IDs.
• The key-value store OUTPUT holds items read and per status, chunks total, upserted, unchanged and deleted (chunksDeleted for chunks left over by shorter items, chunksDeletedRemoved for items no longer in the dataset), embedding tokens (as reported by OpenAI or Cohere), requests and retries, the target, the dimension, and timings per stage (read, chunk, embed, upsert, delete, state, total). It's also written when a run fails partway.
• Without any dataset or keys, the run returns a single row with setup instructions and does nothing else.

How much does it cost to load data into Pinecone or Qdrant?

Vector DB Sync uses pay-per-event pricing: $0.00018 per chunk upserted ($0.18 per 1,000), plus Apify's $0.00005 run start fee. You pay only for chunks that were actually embedded and written to your database. Unchanged chunks, deletions, dry runs, chunks that failed to embed or upsert, and the result rows in the dataset are free (an item marked error is charged only for the chunks of it that were stored). Embedding costs are billed by OpenAI or Cohere to your own key.

Worked example: you crawl 1,000 documentation pages (about 3,000 chunks) every week, and 30 pages change each week.

Examples use the Free-plan rate. Where an example doesn't add it, each run also has its run-start charge, counted per GB of Actor memory (minimum one; one at the default memory); Apify Proxy (where used) is billed separately.

	Chunks embedded and upserted	Vector DB Sync charge
First run	3,000	3,000 × $0.00018 = $0.54
Each weekly run after that (delta mode)	about 90, plus deleting the chunks of pages that got shorter	about $0.016
Each weekly run without delta updates	3,000	$0.54

The other ~2,900 chunks are compared by fingerprint and skipped: no embedding call, no upsert, no charge, and nothing on your OpenAI or Cohere bill.

If you set a maximum cost per run, the Actor checks it before each embedding request and embeds only as many chunks as it can still bill, so you never pay your embedding provider for chunks that weren't stored. The run then stops cleanly, the stopped items say so, and the next run continues where it stopped.

FAQ

The run stopped with a dimension mismatch.

Your index or collection was created with a different vector size than the model produces. Create the Pinecone index with the model's dimension (1536 for text-embedding-3-small, 3072 for -3-large, 1024 for Cohere embed-english-v3.0), or set Dimensions to match your index (OpenAI text-embedding-3 models and Cohere embed-v4.0).

"No Pinecone index at … (HTTP 404)".

Paste the index host shown on the index's page in the Pinecone console (like my-index-abc1234.svc….pinecone.io), not the index name.

Everything was re-embedded, although the site barely changed.

A chunk is re-embedded when its text, its metadata fields or the embedding settings change. Don't pick metadata fields that change on every crawl, like timestamps. Changing the chunk size also re-embeds everything.

"Delete items no longer in the dataset" didn't delete anything.

It needs a Memory name (or a saved task), and for safety it deletes nothing when the run didn't read the whole dataset, when the dataset is empty, when any item had no ID, or when more than half of the remembered items would go. The run log shows the counts; turn on Allow removing more than half if the site really shrank.

My OpenAI quota or Cohere trial limit was hit.

An exhausted OpenAI quota fails the run at once. Cohere trial keys are rate-limited; use a production key for big datasets. Thanks to the memory, the next run skips everything already synced.

Can I use it with LangChain or LlamaIndex?

Yes. The chunk text is stored under text in the metadata (what LangChain and most RAG code expect), next to url and your metadata fields.

Does it work with other vector databases?

Pinecone and Qdrant only.

Limitations

• Memory: about 200 bytes per synced item, so the default 1 GB handles roughly 2 million items per memory. Items with more than 10,000 chunks (about 20 MB of text) are cut at 10,000 chunks.
• What counts as a change: a chunk is re-embedded when its text, its metadata fields, the embedding settings (provider, model, dimensions) or Store the chunk text change (switching the database type changes the metadata size limit, which counts too). Don't pick metadata fields that change on every crawl, like timestamps. Changing the chunk size re-embeds everything and deletes the old chunks. Inserting text near the top of a page shifts its chunks, so the page's later chunks are re-embedded too.
• The memory trusts itself, mostly. It is discarded automatically (with a warning) when the Qdrant collection is missing or has 0 points, or the Pinecone index reports 0 vectors. The Pinecone check uses the whole index's totalVectorCount from describe_index_stats (not the namespace's count), and Pinecone's counts can lag a little behind writes, so a run started seconds after a first sync may re-embed. If you delete only some vectors by hand, turn Delta updates off for one run (or change the memory name) so everything is upserted again. Don't run two syncs with the same memory at the same time.
• OpenAI: up to 2,048 inputs and 300,000 tokens per request, 8,192 tokens per input. Without a tokenizer, the Actor treats every UTF-8 byte as a possible token, so a chunk is cut only if it's over 8,191 bytes (never at the default chunk size). An exhausted quota or spend limit fails the run at once.
• Cohere: up to 96 texts per request. The v3 models read at most 512 tokens per chunk (about 2,000 characters of English) and cut the rest (truncate: END), so keep Chunk size at 2,000 or less with them. Dimensions needs embed-v4.0.
• Pinecone: dense serverless indexes, API version 2026-07; up to 1,000 vectors and 2 MB per upsert, 1,000 IDs per delete, and 40 KB of metadata per vector. Metadata over the limit is shortened (longest values first; the bookkeeping fields are never cut) and counted in chunksMetadataShortened. Metadata values are strings, numbers, booleans or lists of strings, and names never start with $, as Pinecone requires. A delete answered with 404 (namespace already gone) counts as done.
• Qdrant: single unnamed vector or one named vector; collections created by the Actor use Cosine distance. Writes wait until they're applied (wait=true). Up to 256 points per request.
• Retries: up to 7 attempts per request on HTTP 429, 5xx and network errors, waiting at most 90 seconds at a time and 5 minutes in total. A service that asks for a longer wait fails the run, and the next run skips everything already synced.
• URLs with a user name and password (https://user:pass@…) work for Qdrant, but the credentials are removed from logs, OUTPUT and the memory key.
• After the first page, the dataset is read with only the top-level fields the sync uses (text, metadata, ID and url).
• Restarts: if the platform restarts a run, rows already in the output dataset aren't added twice, and chunks already upserted are skipped by the memory. Charges carry an idempotency key per upsert batch.
• Not affiliated with Pinecone, Qdrant, OpenAI or Cohere.

Related Actors

Vector DB Sync is part of a small suite of Actors that work together after any scraper:

Actor	Use it to
Merge & Deduplicate Datasets	Merge several crawls and remove duplicate pages before loading them.
Send Email Report & Slack, Discord, Teams, Telegram Alerts	Get a Slack or email message after each sync.
Google Sheets Sync	Keep a reviewable list of crawled pages in a Google Sheet.
Dataset Delivery	Archive each crawl as a JSONL file on SFTP, Dropbox or Google Drive.
SEC Funding Rounds Tracker	Track newly funded companies from SEC Form D, Form C and Reg A filings.
PDF & Image to Text	Extract text or Markdown from PDFs and scanned documents (with OCR) before loading them for RAG.