Pinecone Integration

Pinecone API KEY

Name of the Pinecone index where the data will be stored

Name of the Pinecone index namespace (partition the records in an index)

Choose the embeddings provider to use for generating embeddings

Configure the parameters for the LangChain embedding class. Key points to consider:

1. Typically, you only need to specify the model name. For example, for OpenAI, set the model name as {"model": "text-embedding-3-small"}.

2. It's required to ensure that the vector size of your embeddings matches the size of embeddings in the database.

3. Here are examples of embedding models:

   • OpenAI: text-embedding-3-small, text-embedding-3-large, etc.
   • Cohere: embed-english-v3.0, embed-multilingual-light-v3.0, etc.

4. For more details about other parameters, refer to the LangChain documentation.

Value of the API KEY for the embeddings provider (if required).

For example for OpenAI it is OPENAI_API_KEY, for Cohere it is COHERE_API_KEY)

This array specifies the dataset fields to be selected and stored in the vector store. Only the fields listed here will be included in the vector store.

For instance, when using the Website Content Crawler, you might choose to include fields such as text, url, and metadata.title in the vector store.

A list of dataset fields which should be selected from the dataset and stored as metadata in the vector stores.

For example, when using the Website Content Crawler, you might want to store url in metadata. In this case, use metadataDatasetFields parameter as follows {"url": "url"}

This object allows you to store custom metadata for every item in the vector store.

For example, if you want to store the domain as metadata, use the metadataObject like this: {"domain": "apify.com"}.

Dataset ID (when running standalone without integration)

Choose the update strategy for the integration. The update strategy determines how the integration updates the data in the database.

The available options are:

• Add data (add):

  • Always adds new records to the database.
  • No checks for existing records or updates are performed.
  • Useful when appending data without concern for duplicates.

• Upsert data (upsert):

  • Updates existing records if they match a key or identifier.
  • Inserts new records into the database if they don't already exist.
  • Ideal for ensuring the database contains the most up-to-date data, avoiding duplicates.

• Update changed data based on deltas (deltaUpdates):

  • Performs incremental updates by identifying differences (deltas) between the new dataset and the existing records.
  • Only adds new records and updates those that have changed.
  • Unchanged records are left untouched.
  • Maximizes efficiency by reducing unnecessary updates.

Select the strategy that best fits your use case.

This array contains fields that are used to uniquely identify dataset items, which helps to handle content changes across different runs.

For instance, in a web content crawling scenario, the url field could serve as a unique identifier for each item.

When set to true, this setting enables incremental updates for objects in the database by comparing the changes (deltas) between the crawled dataset items and the existing objects, uniquely identified by the datasetKeysToItemId field.

The integration will only add new objects and update those that have changed, reducing unnecessary updates. The datasetFields, metadataDatasetFields, and metadataObject fields are used to determine the changes.

This array contains fields that are used to uniquely identify dataset items, which helps to handle content changes across different runs.

For instance, in a web content crawling scenario, the url field could serve as a unique identifier for each item.

When set to true, delete objects from the database that have not been crawled for a specified period.

This setting allows the integration to manage the deletion of objects from the database that have not been crawled for a specified period. It is typically used in subsequent runs after the initial crawl.

When the value is greater than 0, the integration checks if objects have been seen within the last X days (determined by the expiration period). If the objects are expired, they are deleted from the database. The specific value for deletedExpiredObjectsDays depends on your use case and how frequently you crawl data.

For example, if you crawl data daily, you can set deletedExpiredObjectsDays to 7 days. If you crawl data weekly, you can set deletedExpiredObjectsDays to 30 days.

When set to true, the text will be divided into smaller chunks based on the settings provided below. Proper chunking helps optimize retrieval and ensures accurate and efficient responses.

Defines the maximum number of characters in each text chunk. Choosing the right size balances between detailed context and system performance. Optimal sizes ensure high relevancy and minimal response time.

Specifies the number of overlapping characters between consecutive text chunks. Adjusting this helps maintain context across chunks, which is crucial for accuracy in retrieval-augmented generation systems.

When set to true, this option will use Pinecone ID prefix instead of metadata for handling deltaUpdates. It will create a prefix in the database using the following format: item_id#chunk_id. This will results in more efficient updates

The number of texts to embed in a single batch. This setting can be used to optimize the performance when embedding texts. If you receive Embedding provider errors, you need to decrease this size.