# Dataset to Pinecone & Qdrant Vector Export (`oldjard/dataset-to-pinecone-qdrant`) Actor

Send any Apify dataset to Pinecone or Qdrant for RAG and semantic search. Chunks text, embeds with your own OpenAI, Gemini or Cohere key, upserts with stable IDs, skips unchanged records (no re-embedding cost) and can delete stale ones. Runs after any scraper as an integration.

- **URL**: https://apify.com/oldjard/dataset-to-pinecone-qdrant.md
- **Developed by:** [Joshua White](https://apify.com/oldjard) (community)
- **Categories:** AI, Integrations, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Dataset to Pinecone & Qdrant Vector Export

Send any Apify dataset to **Pinecone** or **Qdrant** for RAG, chatbots and semantic search: after a crawler
finishes, on a schedule, or by hand. Text is **chunked**, **embedded with your own OpenAI, Gemini or Cohere key**,
and **upserted with stable IDs**, so a daily run updates records in place instead of piling up duplicates.

On a re-run, records whose text hasn't changed are **skipped before embedding**, so you don't pay your provider to
embed the same page twice. Pages that disappeared can be **deleted** from the index. When something is set up wrong,
the run says which field to fix.

### What you get

- **Pinecone and Qdrant** (Qdrant Cloud or self-hosted), with your own database and keys. Nothing is shared with
  other users, and nothing is stored by this Actor.
- **Your embedding provider:** OpenAI, Google Gemini, Cohere, or any OpenAI-compatible API (Together, Mistral,
  Voyage, a self-hosted server). You pay the provider directly at their price.
- **Matches your index automatically.** The Actor reads the index's dimension and asks the model for that size when
  the model can do it (OpenAI `text-embedding-3-*`, Gemini, Cohere v4). If the sizes can't match, the run stops after
  embedding **one** text and tells you both numbers. Nothing is written.
- **Works on scraper output as-is.** It finds the text in `text`, `markdown`, `content`, `pageContent`, `body` or
  `description` (Website Content Crawler, RAG Web Browser and most scrapers), and identifies items by `url` or `id`.
  You can name the fields instead.
- **Chunking** at paragraph, line or sentence breaks, with overlap. The chunk size and overlap are yours to set.
- **Metadata for filtering:** the item's short fields are kept automatically, or you list the fields you want.
  The chunk text is stored as `text` (LangChain and LlamaIndex read it from there), and values are converted to
  types Pinecone accepts. Records are kept under Pinecone's 40 KB metadata limit.
- **Skip unchanged** (on by default): before embedding, the Actor checks which records the database already holds
  with the same text, metadata and model. Those are skipped and free.
- **Delete stale** (optional): after a complete run, the Actor removes records that this source no longer produces,
  such as removed pages or the extra chunks of a page that got shorter. It never runs after a partial or empty
  run, so a scraper that comes back empty can't wipe your index.
- **Runs as an integration** after any Actor. The finished run's dataset is exported automatically.
- **Retries** rate limits (HTTP 429, honoring `Retry-After`) and server errors, with backoff.
- **Free preview:** with no keys, or with *Preview only* on, you see the first records exactly as they would be
  stored, plus a token estimate, at no cost.

### Quick start

1. **Make the index or collection.**
   - **Pinecone:** in the [Pinecone console](https://app.pinecone.io), create a *dense* serverless index whose
     **dimension** matches your model: 1536 for OpenAI `text-embedding-3-small` (the default). Gemini and Cohere v4
     can fit other sizes, and for those the Actor picks the size automatically. Copy the index **HOST**
     (`https://<index>-<id>.svc.<region>.pinecone.io`) and an API key.
   - **Qdrant:** copy your cluster URL (Qdrant Cloud: `https://<id>.<region>.cloud.qdrant.io:6333`) and an API key
     with write access. You don't need to create the collection; the Actor creates it at the model's size.
2. **Get an embedding key** from [OpenAI](https://platform.openai.com/api-keys),
   [Google AI Studio](https://aistudio.google.com/apikey) or [Cohere](https://dashboard.cohere.com/api-keys).
3. **Run it once by hand** on a dataset: fill in the database fields, the embedding key and **Dataset**. Check the
   summary, then query your index.
4. **Make it automatic:** on your scraper's page open **Integrations > Add integration**, pick this Actor, and use
   the same input. Keep **Dataset** as `{{resource.defaultDatasetId}}`. Turn on **Delete stale records** if the
   index should mirror the source.

All keys are stored encrypted by Apify and never appear in logs, the dataset or error messages.

### Input

| Field | What it does |
|---|---|
| **Dataset** | The dataset to export. As an integration: `{{resource.defaultDatasetId}}`. |
| **Vector database** | `pinecone` (default) or `qdrant`. |
| **Pinecone index host / API key / namespace** | The index HOST from the console, a key, and an optional namespace. |
| **Qdrant URL / API key / collection** | Cluster URL, key (empty for an open self-hosted server), and collection name. |
| **Create the collection if missing**, **Distance**, **Vector name** | Qdrant only. Named-vector collections: say which vector to fill. |
| **Embedding provider / API key / model** | Defaults: `text-embedding-3-small`, `gemini-embedding-001`, `embed-v4.0`. |
| **Embedding dimensions** | Rarely needed. The size is matched to the index automatically when the model allows it. |
| **Embedding API base URL** | OpenAI-compatible APIs only, e.g. `https://api.together.xyz/v1`. |
| **Text fields** | The fields to embed (several are joined). Empty means auto (see above). Nested: `page.body`. |
| **ID fields** | Identify an item across runs. Empty means `url`, else `id`, else the text itself. |
| **Metadata fields** | Fields stored for filtering. Nested `author.name` becomes `author_name`. Empty means short top-level values. |
| **Store the chunk text**, **Metadata key for the text** | On by default, under `text`. |
| **Chunk size / overlap** | In characters. Default 2,000 (~500 tokens) with 200 overlap. |
| **Skip unchanged records** | On by default. Saves embedding cost on re-runs. |
| **Delete stale records** | Off by default. See "How sync works". |
| **Sync key** | Labels the records this export owns. Use one key per source. |
| **Preview only**, **Skip first items**, **Max items** | Limits. A partial export never deletes stale records. |

### How sync works

Every record ID is `<sync key>#<hash of the item's ID fields>#<chunk number>`, for example
`apify#3f2a…#0`. In Qdrant it's a UUID derived from that same string, and the original is kept in the
`apify_id` payload field. The same page always gets the same IDs, so a re-run **updates** its records in place.

Each record also carries `apify_source` (the sync key), `apify_item`, `apify_chunk` and `apify_hash` (a hash of the
text, metadata and model). *Skip unchanged* compares `apify_hash`. *Delete stale* lists every record under the sync
key and deletes the ones this run didn't produce. On Pinecone this uses ID-prefix listing, which only serverless
indexes support. On Qdrant it uses a filter on `apify_source`; the Actor creates the keyword index that Qdrant Cloud
requires for this.

Changing the model or its size changes `apify_hash`, so the next run re-embeds everything. That is what you want,
because vectors from different models don't mix.

### Output

- The run's **dataset** has one summary row: records upserted, unchanged records skipped, stale records deleted,
  items read, chunks, dimension, model, tokens embedded (empty when the provider doesn't report it, as with Gemini's
  `gemini-embedding-001`), and whether the run stopped early. In preview mode the
  dataset holds the first 25 records instead (ID, chunk text, metadata).
- The same summary is saved as `OUTPUT` in the key-value store.

### Errors you might see (and the fix)

| Message starts with | Fix |
|---|---|
| *Pinecone rejected the API key* | Use a key from the project that owns the index. |
| *Pinecone has no index at …* | Copy the HOST again from the index page. A re-created index gets a new host. |
| *…expects 1024-dimensional vectors, but … returned 1536* | Pick a model of that size, set *Embedding dimensions*, or create an index of the model's size. |
| *…rejected the embedding API key* | Check that the key belongs to the selected provider. |
| *…is out of credit or over its quota* | Top up with your provider. Records already written are kept, and the next run skips them. |
| *Google Gemini kept answering "quota exceeded"* | Free-tier Gemini keys have low per-minute and per-day limits. The run waits about a minute before giving up. Run again later; written records are kept. |
| *A chunk was too long* | Lower *Chunk size*. |
| *None of the N items had text to embed* | Put the right field in *Text fields*. The message lists the fields it saw. |
| *Qdrant … has no collection* | Fix the name, or turn on *Create the collection if missing*. |
| *To export to …, also fill in …* | A field is missing. With every database and key field empty, the run is a free preview instead. |
| *This run can't read dataset … because it came only from the integration's event data* | Edit the integration and set **Dataset** to `{{resource.defaultDatasetId}}`. |

### Pricing

Pay per event: **$0.25 per 1,000 records upserted** (one record is one chunk). You pay only for records the database
confirms, so a run that fails midway is charged only for what it wrote. Unchanged records that are skipped, stale
deletions and previews cost nothing. Your embedding provider bills you directly; at OpenAI's
`text-embedding-3-small` price, 1,000 chunks of ~500 tokens cost about $0.01. A run stops cleanly at the maximum
charge you set.

### Limits and notes

- Gemini doesn't reject long text: it embeds only the start. `gemini-embedding-001` reads about 8,000 characters per
  chunk and `gemini-embedding-2` about 32,000. The run warns you if *Chunk size* is larger than that. The default of
  2,000 is well inside both.
- Pinecone: dense indexes; delete stale needs a serverless index. Indexes with integrated embedding (Pinecone hosts
  the model) aren't supported yet. Use a regular dense index.
- Qdrant: dense vectors (unnamed, or one named vector per run); sparse-only collections aren't supported.
- Text only. Images and other files in the dataset aren't embedded.
- Only public datasets you can read and databases you control. The Actor doesn't scrape anything itself.

# Actor input Schema

## `datasetId` (type: `string`):

The dataset to export. Pick it from the list or paste its ID. When this runs as an integration after another Actor, this holds {{resource.defaultDatasetId}}, which Apify replaces with the finished run's dataset. Keep it. Leave everything empty for a free preview on a small sample.

## `target` (type: `string`):

Where the records go.

## `pineconeHost` (type: `string`):

Open your index in the Pinecone console and copy its HOST, e.g. https://my-index-abc1234.svc.aped-4627-b74a.pinecone.io. The index must already exist, with the same dimension as your embedding model (text-embedding-3-small: 1536; the actor asks OpenAI, Gemini and Cohere v4 for the index's size automatically).

## `pineconeApiKey` (type: `string`):

From the Pinecone console, API Keys, in the project that owns the index. Stored encrypted, never logged.

## `pineconeNamespace` (type: `string`):

Optional. Empty uses the index's default namespace.

## `qdrantUrl` (type: `string`):

Your cluster's REST address, e.g. https://xyz-example.eu-central.aws.cloud.qdrant.io:6333 (Qdrant Cloud: cluster details). Self-hosted works too if it's reachable from the internet over https.

## `qdrantApiKey` (type: `string`):

A key with write access (Qdrant Cloud: cluster, API Keys). Empty for a self-hosted Qdrant without auth. Stored encrypted, never logged.

## `qdrantCollection` (type: `string`):

The collection to write to.

## `createCollection` (type: `boolean`):

Create it with the embedding model's size and the distance below.

## `qdrantDistance` (type: `string`):

Used only when the actor creates the collection.

## `qdrantVectorName` (type: `string`):

Only for collections with named vectors: which one to fill. Empty works for unnamed vectors, or when there is exactly one name.

## `embeddingProvider` (type: `string`):

Whose model turns text into vectors. You use your own key and pay the provider directly; the actor never stores or reuses it.

## `embeddingApiKey` (type: `string`):

Your API key for the provider above. Stored encrypted, never logged.

## `embeddingModel` (type: `string`):

Empty uses text-embedding-3-small (OpenAI), gemini-embedding-001 (Gemini) or embed-v4.0 (Cohere). Required for an OpenAI-compatible API.

## `embeddingDimensions` (type: `integer`):

Optional. Normally leave empty: with an existing index the actor asks the model for the index's size when the model allows it (OpenAI text-embedding-3, Gemini, Cohere v4).

## `embeddingBaseUrl` (type: `string`):

Only for an OpenAI-compatible API: its base URL ending in /v1, e.g. https://api.together.xyz/v1.

## `textFields` (type: `array`):

The field(s) whose text is embedded, joined with a blank line. Nested fields use a dot: page.body. Empty picks the first of text, markdown, content, pageContent, body, description (fits Website Content Crawler and most scrapers).

## `idFields` (type: `array`):

The field(s) that identify an item, so a re-run updates its records instead of duplicating them. Empty uses url, else id, else the text itself.

## `metadataFields` (type: `array`):

Fields to store with each record for filtering (nested: author.name becomes author_name). Empty stores the item's short top-level values (strings up to 500 characters, numbers, booleans, short lists).

## `storeText` (type: `boolean`):

Most RAG setups need the text back with each match. On Pinecone it is cut to fit the 40 KB metadata limit if needed (the embedding always uses the whole chunk).

## `textMetadataKey` (type: `string`):

Name of the metadata field that holds the chunk text. LangChain and LlamaIndex default to text.

## `chunkSize` (type: `integer`):

Longer texts are split at paragraph, line or sentence breaks into chunks of at most this many characters (2,000 is about 500 tokens).

## `chunkOverlap` (type: `integer`):

How much each chunk repeats from the end of the previous one, so sentences at a cut aren't lost. At most half the chunk size.

## `skipUnchanged` (type: `boolean`):

Before embedding, check which records the database already has with the same text, metadata and model, and skip them. Saves your embedding bill on daily runs; skipped records are free.

## `deleteStale` (type: `boolean`):

After a complete run, delete records with this sync key that this dataset no longer produces (removed pages, texts that got shorter). Skipped if the run stops early, covers only part of the dataset, or produced nothing, so an empty upstream run never wipes your index. Pinecone: serverless indexes.

## `syncKey` (type: `string`):

Labels the records this export owns (every record ID starts with it). Use a different key per source, e.g. docs-site and blog, so deleting stale records of one never touches the other.

## `dryRun` (type: `boolean`):

Show the first records as they would be stored, and a token estimate, without embedding or writing anything. Free.

## `offset` (type: `integer`):

Start from this item of the dataset.

## `maxItems` (type: `integer`):

Export at most this many items. Empty means all.

## Actor input object example

```json
{
  "target": "pinecone",
  "createCollection": true,
  "qdrantDistance": "Cosine",
  "embeddingProvider": "openai",
  "storeText": true,
  "textMetadataKey": "text",
  "chunkSize": 2000,
  "chunkOverlap": 200,
  "skipUnchanged": true,
  "deleteStale": false,
  "syncKey": "apify",
  "dryRun": false,
  "offset": 0
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("oldjard/dataset-to-pinecone-qdrant").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("oldjard/dataset-to-pinecone-qdrant").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call oldjard/dataset-to-pinecone-qdrant --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,oldjard/dataset-to-pinecone-qdrant"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/w5ggaZ1BNdNobjYzF/builds/LmmLAPDLwTryReFZF/openapi.json