No credit card required
Pinecone integration
No credit card required
Simplify your data operations with this Apify and Pinecone integration. Easily push selected fields from your Apify Actor directly into any Pinecone index. If the index doesn't exist, the integration will create it. Practical and straightforward solution for handling data between Apify and Pinecone.
integration failing to create index
Closed
No matter what I do the integration always fails on "creating index" after an actor run triggers the integration.
Hello, I've just released new version of integration with extended logging. Feel free to try it out and let me know.
wobbleburger
This is still not working, here is where it fails:
2023-07-26T01:28:36.084Z Pinecone initialized 2023-07-26T01:28:36.135Z Creating index 2023-07-26T01:28:38.843Z Index creation failed: Incorrect API key provided: hIrxaf-7********sgyg. You can find your API key at https://platform.openai.com/account/api-keys. 2023-07-26T01:28:38.859Z INFO Exiting actor ({"exit_code": 1})
The problem is, the API key that it shows is not my API key for Pinecone OR for OpenAI...
wobbleburger
Ok, I got past that by putting in my credentials again. Here is the new error I am getting:
2023-07-26T01:33:06.167Z ACTOR: Pulling Docker image from repository. 2023-07-26T01:33:12.641Z ACTOR: Creating Docker container. 2023-07-26T01:33:12.835Z ACTOR: Starting Docker container. 2023-07-26T01:33:16.268Z INFO Initializing actor... 2023-07-26T01:33:16.270Z INFO System info ({"apify_sdk_version": "1.1.1", "apify_client_version": "1.3.0", "python_version": "3.11.3", "os": "linux"}) 2023-07-26T01:33:16.483Z Loading dataset 2023-07-26T01:33:16.500Z Dataset loaded for field text 2023-07-26T01:33:16.502Z Loading documents for field text 2023-07-26T01:33:16.585Z Documents loaded 2023-07-26T01:33:16.587Z Splitting documents 2023-07-26T01:33:16.590Z Documents split 2023-07-26T01:33:16.592Z Initializing pinecone 2023-07-26T01:33:17.301Z Pinecone initialized 2023-07-26T01:33:17.357Z Creating index 2023-07-26T01:33:21.172Z Index created 2023-07-26T01:33:21.195Z Dataset loaded for field crawl 2023-07-26T01:33:21.197Z Loading documents for field crawl 2023-07-26T01:33:21.297Z ERROR Actor failed with an exception 2023-07-26T01:33:21.300Z Traceback (most recent call last): 2023-07-26T01:33:21.302Z File "/usr/src/app/src/main.py", line 34, in main 2023-07-26T01:33:21.304Z documents = loader.load() 2023-07-26T01:33:21.307Z ^^^^^^^^^^^^^ 2023-07-26T01:33:21.309Z File "/opt/venv/lib/python3.11/site-packages/langchain/document_loaders/apify_dataset.py", line 54, in load 2023-07-26T01:33:21.311Z return list(map(self.dataset_mapping_function, dataset_items)) 2023-07-26T01:33:21.314Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2023-07-26T01:33:21.316Z File "/usr/src/app/src/main.py", line 27, in
This could be related with the fact that your selected field is not string type. Try to select string field instead and let me know :)
wobbleburger
Thanks, that works but I need to get some of the fields that are nested within the Apify schema, such as crawl.loaded_url. Also, I need to push a hardcoded field into Pinecone called "source". Every vector I send has a metadata key called "source" with value "website".
Was thinking you could add support for arbitrary "key": "value" where key could be the name of the metadata and value could be a fixed string or a json path?
Ok, I've come up with 2 new optional fields for metadata:
metadata_values- Object of metadata values you want to push to Pinecone from your Actor. For example, if you want to pushurlandcreatedAtvalues to Pinecone, you should set this field to{"url": "https://www.apify.com", "createdAt": "2021-09-01"}.metadata_fields- Object of metadata fields you want to push to Pinecone from your Actor. For example, if you want to pushurlandcreatedAtfields, you should set this field to{"url": "url", "createdAt": "createdAt"}. If it has the same key asmetadata_values, it's replaced.
Feel free to test it. I'm closing this issue, feel free to reopen if necessary.
Jan Turoň- 17 monthly users
- 99.7% runs succeeded
- 11 days response time
- Created in May 2023
- Modified 5 days ago
