![Pinecone Integration avatar](https://images.apifyusercontent.com/kYb1zzP0YvosRaXbVq5Jc8Vt-jN1kOF207VoXrxouQY/rs:fill:92:92/aHR0cHM6Ly9hcGlmeS1pbWFnZS11cGxvYWRzLXByb2QuczMudXMtZWFzdC0xLmFtYXpvbmF3cy5jb20vVkRDMGpkcTUxNjh5V2V6cEMvRThkM2NqdWRvM2xmTERVODUtaW1hZ2VzLnBuZw.webp)
No credit card required
![Pinecone Integration](https://images.apifyusercontent.com/kYb1zzP0YvosRaXbVq5Jc8Vt-jN1kOF207VoXrxouQY/rs:fill:92:92/aHR0cHM6Ly9hcGlmeS1pbWFnZS11cGxvYWRzLXByb2QuczMudXMtZWFzdC0xLmFtYXpvbmF3cy5jb20vVkRDMGpkcTUxNjh5V2V6cEMvRThkM2NqdWRvM2xmTERVODUtaW1hZ2VzLnBuZw.webp)
Pinecone Integration
No credit card required
This integration transfers data from Apify Actors to a Pinecone and is a good starting point for a question-answering, search, or RAG use case.
integration failing to create index
Closed
No matter what I do the integration always fails on "creating index" after an actor run triggers the integration.
![jan.turon avatar](https://apify-image-uploads-prod.s3.amazonaws.com/2dD9ZC8gfGi6LCjqD/TQRDKWLwLL6sLdscE-DSCF6206_edit_%281%29ff.jpeg)
Hello, I've just released new version of integration with extended logging. Feel free to try it out and let me know.
wobbleburger
This is still not working, here is where it fails:
2023-07-26T01:28:36.084Z Pinecone initialized 2023-07-26T01:28:36.135Z Creating index 2023-07-26T01:28:38.843Z Index creation failed: Incorrect API key provided: hIrxaf-7********sgyg. You can find your API key at https://platform.openai.com/account/api-keys. 2023-07-26T01:28:38.859Z INFO Exiting actor ({"exit_code": 1})
The problem is, the API key that it shows is not my API key for Pinecone OR for OpenAI...
wobbleburger
Ok, I got past that by putting in my credentials again. Here is the new error I am getting:
2023-07-26T01:33:06.167Z ACTOR: Pulling Docker image from repository. 2023-07-26T01:33:12.641Z ACTOR: Creating Docker container. 2023-07-26T01:33:12.835Z ACTOR: Starting Docker container. 2023-07-26T01:33:16.268Z INFO Initializing actor... 2023-07-26T01:33:16.270Z INFO System info ({"apify_sdk_version": "1.1.1", "apify_client_version": "1.3.0", "python_version": "3.11.3", "os": "linux"}) 2023-07-26T01:33:16.483Z Loading dataset 2023-07-26T01:33:16.500Z Dataset loaded for field text 2023-07-26T01:33:16.502Z Loading documents for field text 2023-07-26T01:33:16.585Z Documents loaded 2023-07-26T01:33:16.587Z Splitting documents 2023-07-26T01:33:16.590Z Documents split 2023-07-26T01:33:16.592Z Initializing pinecone 2023-07-26T01:33:17.301Z Pinecone initialized 2023-07-26T01:33:17.357Z Creating index 2023-07-26T01:33:21.172Z Index created 2023-07-26T01:33:21.195Z Dataset loaded for field crawl 2023-07-26T01:33:21.197Z Loading documents for field crawl 2023-07-26T01:33:21.297Z ERROR Actor failed with an exception 2023-07-26T01:33:21.300Z Traceback (most recent call last): 2023-07-26T01:33:21.302Z File "/usr/src/app/src/main.py", line 34, in main 2023-07-26T01:33:21.304Z documents = loader.load() 2023-07-26T01:33:21.307Z ^^^^^^^^^^^^^ 2023-07-26T01:33:21.309Z File "/opt/venv/lib/python3.11/site-packages/langchain/document_loaders/apify_dataset.py", line 54, in load 2023-07-26T01:33:21.311Z return list(map(self.dataset_mapping_function, dataset_items)) 2023-07-26T01:33:21.314Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2023-07-26T01:33:21.316Z File "/usr/src/app/src/main.py", line 27, in
![jan.turon avatar](https://apify-image-uploads-prod.s3.amazonaws.com/2dD9ZC8gfGi6LCjqD/TQRDKWLwLL6sLdscE-DSCF6206_edit_%281%29ff.jpeg)
This could be related with the fact that your selected field is not string type. Try to select string field instead and let me know :)
wobbleburger
Thanks, that works but I need to get some of the fields that are nested within the Apify schema, such as crawl.loaded_url. Also, I need to push a hardcoded field into Pinecone called "source". Every vector I send has a metadata key called "source" with value "website".
Was thinking you could add support for arbitrary "key": "value" where key could be the name of the metadata and value could be a fixed string or a json path?
![jan.turon avatar](https://apify-image-uploads-prod.s3.amazonaws.com/2dD9ZC8gfGi6LCjqD/TQRDKWLwLL6sLdscE-DSCF6206_edit_%281%29ff.jpeg)
Ok, I've come up with 2 new optional fields for metadata:
metadata_values
- Object of metadata values you want to push to Pinecone from your Actor. For example, if you want to pushurl
andcreatedAt
values to Pinecone, you should set this field to{"url": "https://www.apify.com", "createdAt": "2021-09-01"}
.metadata_fields
- Object of metadata fields you want to push to Pinecone from your Actor. For example, if you want to pushurl
andcreatedAt
fields, you should set this field to{"url": "url", "createdAt": "createdAt"}
. If it has the same key asmetadata_values
, it's replaced.
Feel free to test it. I'm closing this issue, feel free to reopen if necessary.
- 18 monthly users
- 3 stars
- 75.0% runs succeeded
- 1.3 days response time
- Created in Jun 2024
- Modified 2 days ago