Pinecone integration avatar
Pinecone integration
Try for free

No credit card required

View all Actors
Pinecone integration

Pinecone integration

jan.turon/pinecone-integration
Try for free

No credit card required

Simplify your data operations with this Apify and Pinecone integration. Easily push selected fields from your Apify Actor directly into any Pinecone index. If the index doesn't exist, the integration will create it. Practical and straightforward solution for handling data between Apify and Pinecone.

User avatar

integration failing to create index

Closed

wobbleburger opened this issue
10 months ago

No matter what I do the integration always fails on "creating index" after an actor run triggers the integration.

User avatar

Hello, I've just released new version of integration with extended logging. Feel free to try it out and let me know.

User avatar

wobbleburger

10 months ago

This is still not working, here is where it fails:

2023-07-26T01:28:36.084Z Pinecone initialized 2023-07-26T01:28:36.135Z Creating index 2023-07-26T01:28:38.843Z Index creation failed: Incorrect API key provided: hIrxaf-7********sgyg. You can find your API key at https://platform.openai.com/account/api-keys. 2023-07-26T01:28:38.859Z INFO Exiting actor ({"exit_code": 1})

The problem is, the API key that it shows is not my API key for Pinecone OR for OpenAI...

User avatar

wobbleburger

10 months ago

Ok, I got past that by putting in my credentials again. Here is the new error I am getting:

2023-07-26T01:33:06.167Z ACTOR: Pulling Docker image from repository. 2023-07-26T01:33:12.641Z ACTOR: Creating Docker container. 2023-07-26T01:33:12.835Z ACTOR: Starting Docker container. 2023-07-26T01:33:16.268Z INFO Initializing actor... 2023-07-26T01:33:16.270Z INFO System info ({"apify_sdk_version": "1.1.1", "apify_client_version": "1.3.0", "python_version": "3.11.3", "os": "linux"}) 2023-07-26T01:33:16.483Z Loading dataset 2023-07-26T01:33:16.500Z Dataset loaded for field text 2023-07-26T01:33:16.502Z Loading documents for field text 2023-07-26T01:33:16.585Z Documents loaded 2023-07-26T01:33:16.587Z Splitting documents 2023-07-26T01:33:16.590Z Documents split 2023-07-26T01:33:16.592Z Initializing pinecone 2023-07-26T01:33:17.301Z Pinecone initialized 2023-07-26T01:33:17.357Z Creating index 2023-07-26T01:33:21.172Z Index created 2023-07-26T01:33:21.195Z Dataset loaded for field crawl 2023-07-26T01:33:21.197Z Loading documents for field crawl 2023-07-26T01:33:21.297Z ERROR Actor failed with an exception 2023-07-26T01:33:21.300Z Traceback (most recent call last): 2023-07-26T01:33:21.302Z File "/usr/src/app/src/main.py", line 34, in main 2023-07-26T01:33:21.304Z documents = loader.load() 2023-07-26T01:33:21.307Z ^^^^^^^^^^^^^ 2023-07-26T01:33:21.309Z File "/opt/venv/lib/python3.11/site-packages/langchain/document_loaders/apify_dataset.py", line 54, in load 2023-07-26T01:33:21.311Z return list(map(self.dataset_mapping_function, dataset_items)) 2023-07-26T01:33:21.314Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2023-07-26T01:33:21.316Z File "/usr/src/app/src/main.py", line 27, in

User avatar

This could be related with the fact that your selected field is not string type. Try to select string field instead and let me know :)

User avatar

wobbleburger

10 months ago

Thanks, that works but I need to get some of the fields that are nested within the Apify schema, such as crawl.loaded_url. Also, I need to push a hardcoded field into Pinecone called "source". Every vector I send has a metadata key called "source" with value "website".

Was thinking you could add support for arbitrary "key": "value" where key could be the name of the metadata and value could be a fixed string or a json path?

User avatar

Ok, I've come up with 2 new optional fields for metadata:

  • metadata_values - Object of metadata values you want to push to Pinecone from your Actor. For example, if you want to push url and createdAt values to Pinecone, you should set this field to {"url": "https://www.apify.com", "createdAt": "2021-09-01"}.
  • metadata_fields - Object of metadata fields you want to push to Pinecone from your Actor. For example, if you want to push url and createdAt fields, you should set this field to {"url": "url", "createdAt": "createdAt"}. If it has the same key as metadata_values, it's replaced.

Feel free to test it. I'm closing this issue, feel free to reopen if necessary.

Developer
Maintained by Community
Actor metrics
  • 15 monthly users
  • 84.2% runs succeeded
  • 11 days response time
  • Created in May 2023
  • Modified 14 days ago