Dataset to Pinecone & Qdrant Vector Export

Send any Apify dataset to Pinecone or Qdrant for RAG, chatbots and semantic search: after a crawler finishes, on a schedule, or by hand. Text is chunked, embedded with your own OpenAI, Gemini or Cohere key, and upserted with stable IDs, so a daily run updates records in place instead of piling up duplicates.

On a re-run, records whose text hasn't changed are skipped before embedding, so you don't pay your provider to embed the same page twice. Pages that disappeared can be deleted from the index. When something is set up wrong, the run says which field to fix.

What you get

• Pinecone and Qdrant (Qdrant Cloud or self-hosted), with your own database and keys. Nothing is shared with other users, and nothing is stored by this Actor.
• Your embedding provider: OpenAI, Google Gemini, Cohere, or any OpenAI-compatible API (Together, Mistral, Voyage, a self-hosted server). You pay the provider directly at their price.
• Matches your index automatically. The Actor reads the index's dimension and asks the model for that size when the model can do it (OpenAI text-embedding-3-*, Gemini, Cohere v4). If the sizes can't match, the run stops after embedding one text and tells you both numbers. Nothing is written.
• Works on scraper output as-is. It finds the text in text, markdown, content, pageContent, body or description (Website Content Crawler, RAG Web Browser and most scrapers), and identifies items by url or id. You can name the fields instead.
• Chunking at paragraph, line or sentence breaks, with overlap. The chunk size and overlap are yours to set.
• Metadata for filtering: the item's short fields are kept automatically, or you list the fields you want. The chunk text is stored as text (LangChain and LlamaIndex read it from there), and values are converted to types Pinecone accepts. Records are kept under Pinecone's 40 KB metadata limit.
• Skip unchanged (on by default): before embedding, the Actor checks which records the database already holds with the same text, metadata and model. Those are skipped and free.
• Delete stale (optional): after a complete run, the Actor removes records that this source no longer produces, such as removed pages or the extra chunks of a page that got shorter. It never runs after a partial or empty run, so a scraper that comes back empty can't wipe your index.
• Runs as an integration after any Actor. The finished run's dataset is exported automatically.
• Retries rate limits (HTTP 429, honoring Retry-After) and server errors, with backoff.
• Free preview: with no keys, or with Preview only on, you see the first records exactly as they would be stored, plus a token estimate, at no cost.

Quick start

1. Make the index or collection.
   • Pinecone: in the Pinecone console, create a dense serverless index whose dimension matches your model: 1536 for OpenAI text-embedding-3-small (the default). Gemini and Cohere v4 can fit other sizes, and for those the Actor picks the size automatically. Copy the index HOST (https://<index>-<id>.svc.<region>.pinecone.io) and an API key.
   • Qdrant: copy your cluster URL (Qdrant Cloud: https://<id>.<region>.cloud.qdrant.io:6333) and an API key with write access. You don't need to create the collection; the Actor creates it at the model's size.
2. Get an embedding key from OpenAI, Google AI Studio or Cohere.
3. Run it once by hand on a dataset: fill in the database fields, the embedding key and Dataset. Check the summary, then query your index.
4. Make it automatic: on your scraper's page open Integrations > Add integration, pick this Actor, and use the same input. Keep Dataset as {{resource.defaultDatasetId}}. Turn on Delete stale records if the index should mirror the source.

All keys are stored encrypted by Apify and never appear in logs, the dataset or error messages.

Input

Field	What it does
Dataset	The dataset to export. As an integration: {{resource.defaultDatasetId}}.
Vector database	pinecone (default) or qdrant.
Pinecone index host / API key / namespace	The index HOST from the console, a key, and an optional namespace.
Qdrant URL / API key / collection	Cluster URL, key (empty for an open self-hosted server), and collection name.
Create the collection if missing, Distance, Vector name	Qdrant only. Named-vector collections: say which vector to fill.
Embedding provider / API key / model	Defaults: text-embedding-3-small, gemini-embedding-001, embed-v4.0.
Embedding dimensions	Rarely needed. The size is matched to the index automatically when the model allows it.
Embedding API base URL	OpenAI-compatible APIs only, e.g. https://api.together.xyz/v1.
Text fields	The fields to embed (several are joined). Empty means auto (see above). Nested: page.body.
ID fields	Identify an item across runs. Empty means url, else id, else the text itself.
Metadata fields	Fields stored for filtering. Nested author.name becomes author_name. Empty means short top-level values.
Store the chunk text, Metadata key for the text	On by default, under text.
Chunk size / overlap	In characters. Default 2,000 (~500 tokens) with 200 overlap.
Skip unchanged records	On by default. Saves embedding cost on re-runs.
Delete stale records	Off by default. See "How sync works".
Sync key	Labels the records this export owns. Use one key per source.
Preview only, Skip first items, Max items	Limits. A partial export never deletes stale records.

How sync works

Every record ID is <sync key>#<hash of the item's ID fields>#<chunk number>, for example apify#3f2a…#0. In Qdrant it's a UUID derived from that same string, and the original is kept in the apify_id payload field. The same page always gets the same IDs, so a re-run updates its records in place.

Each record also carries apify_source (the sync key), apify_item, apify_chunk and apify_hash (a hash of the text, metadata and model). Skip unchanged compares apify_hash. Delete stale lists every record under the sync key and deletes the ones this run didn't produce. On Pinecone this uses ID-prefix listing, which only serverless indexes support. On Qdrant it uses a filter on apify_source; the Actor creates the keyword index that Qdrant Cloud requires for this.

Changing the model or its size changes apify_hash, so the next run re-embeds everything. That is what you want, because vectors from different models don't mix.

Output

• The run's dataset has one summary row: records upserted, unchanged records skipped, stale records deleted, items read, chunks, dimension, model, tokens embedded (empty when the provider doesn't report it, as with Gemini's gemini-embedding-001), and whether the run stopped early. In preview mode the dataset holds the first 25 records instead (ID, chunk text, metadata).
• The same summary is saved as OUTPUT in the key-value store.

Errors you might see (and the fix)

Message starts with	Fix
Pinecone rejected the API key	Use a key from the project that owns the index.
Pinecone has no index at …	Copy the HOST again from the index page. A re-created index gets a new host.
…expects 1024-dimensional vectors, but … returned 1536	Pick a model of that size, set Embedding dimensions, or create an index of the model's size.
…rejected the embedding API key	Check that the key belongs to the selected provider.
…is out of credit or over its quota	Top up with your provider. Records already written are kept, and the next run skips them.
Google Gemini kept answering "quota exceeded"	Free-tier Gemini keys have low per-minute and per-day limits. The run waits about a minute before giving up. Run again later; written records are kept.
A chunk was too long	Lower Chunk size.
None of the N items had text to embed	Put the right field in Text fields. The message lists the fields it saw.
Qdrant … has no collection	Fix the name, or turn on Create the collection if missing.
To export to …, also fill in …	A field is missing. With every database and key field empty, the run is a free preview instead.
This run can't read dataset … because it came only from the integration's event data	Edit the integration and set Dataset to {{resource.defaultDatasetId}}.

Pricing

Pay per event: $0.25 per 1,000 records upserted (one record is one chunk). You pay only for records the database confirms, so a run that fails midway is charged only for what it wrote. Unchanged records that are skipped, stale deletions and previews cost nothing. Your embedding provider bills you directly; at OpenAI's text-embedding-3-small price, 1,000 chunks of ~500 tokens cost about $0.01. A run stops cleanly at the maximum charge you set.

Limits and notes

• Gemini doesn't reject long text: it embeds only the start. gemini-embedding-001 reads about 8,000 characters per chunk and gemini-embedding-2 about 32,000. The run warns you if Chunk size is larger than that. The default of 2,000 is well inside both.
• Pinecone: dense indexes; delete stale needs a serverless index. Indexes with integrated embedding (Pinecone hosts the model) aren't supported yet. Use a regular dense index.
• Qdrant: dense vectors (unnamed, or one named vector per run); sparse-only collections aren't supported.
• Text only. Images and other files in the dataset aren't embedded.
• Only public datasets you can read and databases you control. The Actor doesn't scrape anything itself.