Pinecone suddenly failing

It even fails, when i login to another apify account and use a new pinecone.

My mistake, pinecone git a limit

No, it still a problem, i get error 429. It sasy read over 2000 per second. But it is just weird, because all my flows where working a few days ago.

I think if it could be able to make it read a bit slower, so I won't hit the limit, sometimes it works, but rarely.

Hi, thank you for using the pinecone integration!

First and foremost, I apologize for the delayed response; I missed the notification about the issue.

I’m currently reviewing your run but haven’t identified any obvious issues so far. There is a rate limit of 2,000 query read units per second per index (set by Pinecone), which the integration should typically not exceed, but it appears that it is happening.

I'll work on reproducing the issue and aim to have it resolved today or by tomorrow at the latest.

Okay great. All my pinecone integrations i have made have started to fail the last few days. I can send a lot more example if you need. I seems like the first one I push to pinecone, if I make a new pinecone account works, but the second time and thereafter it fails.

I've tested your setup, and everything seems to be working on my end. I still find it quite unlikely that we would hit the Pinecone rate limit (2k queries per second).

To help us track the issue, I’ve introduced a debug log. I’ve built a beta release that includes the following log:

1for (k, item_id) in enumerate(items_ids):
2    if k % 100 == 0:
3        Actor.log.info("Processing item_id %s (%s/%s) to compare crawled data with the database", item_id, k, len(items_ids))
4    crawled_db[item_id] = vector_store.get_by_item_id(item_id)

If you could test it on your end, I’d greatly appreciate it. You can switch to the new build (tag) in the integration settings under Run Options. At the very least, this will give us more insight into what's happening.

Depending on what we find, I’m considering implementing a retry mechanism with an exponential back-off or a similar approach. I'll do that tomorrow.

See this run id: 8Df5kbl3PBGP95lte

2024-09-29T17:32:47.650Z ACTOR: Pulling Docker image of build mQYdkV7ftThCjypJn from repository. 2024-09-29T17:33:01.384Z ACTOR: Creating Docker container. 2024-09-29T17:33:01.458Z ACTOR: Starting Docker container. 2024-09-29T17:33:04.611Z INFO Initializing actor... 2024-09-29T17:33:04.614Z INFO System info ({"apify_sdk_version": "1.7.2", "apify_client_version": "1.6.4", "python_version": "3.11.10", "os": "linux"}) 2024-09-29T17:33:04.619Z INFO Starting the Vector Store Actor 2024-09-29T17:33:04.763Z INFO Received start argument (vector database name): pinecone 2024-09-29T17:33:04.764Z INFO Get embeddings class: OpenAI 2024-09-29T17:33:05.681Z INFO Load Dataset ID dBbM1MMyBsdGfGhQx and extract fields ['Product URL', 'Titel', 'Pris', 'Pris_før_rabat', 'Image URL', 'Lagerstatus', 'Tilbud', 'Kort_beskrivelse', 'Yderligere_information', 'Beskrivelse'] 2024-09-29T17:33:07.369Z INFO Dataset loaded, number of documents: 1410 2024-09-29T17:33:07.601Z INFO Documents chunked to 1411 chunks 2024-09-29T17:33:08.467Z INFO Update database with crawled data. Delta updates enabled 2024-09-29T17:33:08.468Z INFO Comparing crawled data with the database ... 2024-09-29T17:33:10.175Z ERROR (429) 2024-09-29T17:33:10.176Z Reason: Too Many Requests 2024-09-29T17:33:10.178Z HTTP response headers: HTTPHeaderDict({'Date': 'Sun, 29 Sep 2024 17:33:10 GMT', 'Content-Type': 'application/json', 'Content-Length': '168', 'Connection': 'keep-alive', 'x-pinecone-request-latency-ms': '0', 'x-pinecone-request-id': '4539119554456703513', 'x-envoy-upstream-service-time': '1', 'server': 'envoy'}) 2024-09-29T17:33:10.179Z HTTP response body: {"code":8,"message":"Request failed. You've reached the max query read units per second for index hhspro (2000 read units per second). Pace your queries.","details":[]} 2024-09-29T17:33:10.180Z 2024-09-29T17:33:10.181Z ERROR Actor failed with an exception 2024-09-29T17:33:10.182Z Traceback (most recent call last): 2024-09-29T17:33:10.183Z File "/usr/src/app/src/main.py", line 69, in run_actor 2024-09-29T17:33:10.185Z update_db_with_crawled_data(vcs_, documents) 2024-09-29T17:33:10.186Z File "/usr/src/app/src/vcs.py", line 60, in update_db_with_crawled_data 2024-09-29T17:33:10.187Z data_add, ids_update_last_seen, ids_del = compare_crawled_data_with_db(vector_store, documents) 2024-09-29T17:33:10.189Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.190Z File "/usr/src/app/src/vcs.py", line 99, in compare_crawled_data_with_db 2024-09-29T17:33:10.191Z crawled_db = {item_id: vector_store.get_by_item_id(item_id) for item_id in {d.metadata["item_id"] for d in data}} 2024-09-29T17:33:10.192Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.194Z File "/usr/src/app/src/vcs.py", line 99, in 2024-09-29T17:33:10.195Z crawled_db = {item_id: vector_store.get_by_item_id(item_id) for item_id in {d.metadata["item_id"] for d in data}} 2024-09-29T17:33:10.196Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.198Z File "/usr/src/app/src/vector_stores/pinecone.py", line 43, in get_by_item_id 2024-09-29T17:33:10.199Z results = self.index.query( 2024-09-29T17:33:10.200Z ^^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.202Z File "/usr/local/lib/python3.11/site-packages/pinecone/utils/error_handling.py", line 11, in inner_func 2024-09-29T17:33:10.203Z return func(*args, **kwargs) 2024-09-29T17:33:10.204Z ^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.206Z File "/usr/local/lib/python3.11/site-packages/pinecone/data/index.py", line 429, in query 2024-09-29T17:33:10.207Z response = self._vector_api.query( 2024-09-29T17:33:10.208Z ^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.209Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/api_client.py", line 761, in call 2024-09-29T17:33:10.211Z return self.callable(self, *args, **kwargs) 2024-09-29T17:33:10.212Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.213Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/data/api/data_plane_api.py", line 639, in __query 2024-09-29T17:33:10.214Z return self.call_with_http_info(**kwargs) 2024-09-29T17:33:10.215Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.217Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/api_client.py", line 819, in call_with_http_info 2024-09-29T17:33:10.218Z return self.api_client.call_api( 2024-09-29T17:33:10.219Z ^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.220Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/api_client.py", line 380, in call_api 2024-09-29T17:33:10.221Z return self.__call_api( 2024-09-29T17:33:10.222Z ^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.224Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/api_client.py", line 187, in __call_api 2024-09-29T17:33:10.225Z raise e 2024-09-29T17:33:10.226Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/api_client.py", line 175, in _call_api 2024-09-29T17:33:10.229Z response_data = self.request( 2024-09-29T17:33:10.230Z ^^^^^^^^^^^^^ 2024-09-29T17:33:10.231Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/api_client.py", line 460, in request 2024-09-29T17:33:10.233Z return self.rest_client.POST( 2024-09-29T17:33:10.234Z ^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.235Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/rest.py", line 345, in POST 2024-09-29T17:33:10.236Z return self.request( 2024-09-29T17:33:10.237Z ^^^^^^^^^^^^^ 2024-09-29T17:33:10.238Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/rest.py", line 279, in request 2024-09-29T17:33:10.239Z raise PineconeApiException(http_resp=r) 2024-09-29T17:33:10.240Z pinecone.core.openapi.shared.exceptions.PineconeApiException: (429) 2024-09-29T17:33:10.241Z Reason: Too Many Requests 2024-09-29T17:33:10.243Z HTTP response headers: HTTPHeaderDict({'Date': 'Sun, 29 Sep 2024 17:33:10 GMT', 'Content-Type': 'application/json', 'Content-Length': '168', 'Connection': 'keep-alive', 'x-pinecone-request-latency-ms': '0', 'x-pinecone-request-id': '4539119554456703513', 'x-envoy-upstream-service-time': '1', 'server': 'envoy'}) 2024-09-29T17:33:10.244Z HTTP response body: {"code":8,"message":"Request failed. You've reached the max query read units per second for index hhspro (2000 read units per second). Pace your queries.","details":[]} 2024-09-29T17:33:10.245Z INFO Exiting actor ({"exit_code": 1}) 2024-09-29T17:32:47.650Z ACTOR: Pulling Docker image of build mQYdkV7ftThCjypJn from repository. 2024-09-29T17:33:01.384Z ACTOR: Creating Docker container. 2024-09-29T17:33:01.458Z ACTOR: Starting Docker container. 2024-09-29T17:33:04.611Z INFO Initializing actor... 2024-09-29T17:33:04.614Z INFO System info ({"apify_sdk_version": "1.7.2", "apify_client_version": "1.6.4", "python_version": "3.11.10", "os": "linux"}) 2024-09-29T17:33:04.619Z INFO Starting the Vector Store Actor 2024-09-29T17:33:04.763Z INFO Received start argument (vector database name): pinecone 2024-09-29T17:33:04.764Z INFO Get embeddings class: OpenAI 2024-09-29T17:33:05.681Z INFO Load Dataset ID dBbM1MMyBsdGfGhQx and extract fields ['Product URL', 'Titel', 'Pris', 'Pris_før_rabat', 'Image URL', 'Lagerstatus', 'Tilbud', 'Kort_beskrivelse', 'Yderligere_information', 'Beskrivelse'] 2024-09-29T17:33:07.369Z INFO Dataset loaded, number of documents: 1410 2024-09-29T17:33:07.601Z INFO Documents chunked to 1411 chunks 2024-09-29T17:33:08.467Z INFO Update database with crawled data. Delta updates enabled 2024-09-29T17:33:08.468Z INFO Comparing crawled data with the database ... 2024-09-29T17:33:10.175Z ERROR (429) 2024-09-29T17:33:10.176Z Reason: Too Many Requests 2024-09-29T17:33:10.178Z HTTP response headers: HTTPHeaderDict({'Date': 'Sun, 29 Sep 2024 17:33:10 GMT', 'Content-Type': 'application/json', 'Content-Length': '168', 'Connection': 'keep-alive', 'x-pinecone-request-latency-ms': '0', 'x-pinecone-request-id': '4539119554456703513', 'x-envoy-upstream-service-time': '1', 'server': 'envoy'}) 2024-09-29T17:33:10.179Z HTTP response body: {"code":8,"message":"Request failed. You've reached the max query read units per second for index hhspro (2000 read units per second). Pace your queries.","details":[]} 2024-09-29T17:33:10.180Z 2024-09-29T17:33:10.181Z ERROR Actor failed with an exception 2024-09-29T17:33:10.182Z Traceback (most recent call last): 2024-09-29T17:33:10.183Z File "/usr/src/app/src/main.py", line 69, in run_actor 2024-09-29T17:33:10.185Z update_db_with_crawled_data(vcs, documents) 2024-09-29T17:33:10.186Z File "/usr/src/app/src/vcs.py", line 60, in update_db_with_crawled_data 2024-09-29T17:33:10.187Z data_add, ids_update_last_seen, ids_del = compare_crawled_data_with_db(vector_store, documents) 2024-09-29T17:33:10.189Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.190Z File "/usr/src/app/src/vcs.py", line 99, in compare_crawled_data_with_db 2024-09-29T17:33:10.191Z crawled_db = {item_id: vector_store.get_by_item_id(item_id) for item_id in {d.metadata["item_id"] for d in data}} 2024-09-29T17:33:10.192Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.194Z File "/usr/src/app/src/vcs.py", line 99, in 2024-09-29T17:33:10.195Z crawled_db = {item_id: vector_store.get_by_item_id(item_id) for item_id in {d.metadata["item_id"] for d in data}} 2024-09-29T17:33:10.196Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.198Z File "/usr/src/app/src/vector_stores/pinecone.py", line 43, in get_by_item_id 2024-09-29T17:33:10.199Z results = self.index.query( 2024-09-29T17:33:10.200Z ^^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.202Z File "/usr/local/lib/python3.11/site-packages/pinecone/utils/error_handling.py", line 11, in inner_func 2024-09-29T17:33:10.203Z return func(*args, **kwargs) 2024-09-29T17:33:10.204Z ^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.206Z File "/usr/local/lib/python3.11/site-packages/pinecone/data/index.py", line 429, in query 2024-09-29T17:33:10.207Z response = self._vector_api.query( 2024-09-29T17:33:10.208Z ^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.209Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/api_client.py", line 761, in call 2024-09-29T17:33:10.211Z return self.callable(self, *args, **kwargs) 2024-09-29T17:33:10.212Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.213Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/data/api/data_plane_api.py", line 639, in __query 2024-09-29T17:33:10.214Z return self.call_with_http_info(**kwargs) 2024-09-29T17:33:10.215Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.217Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/api_client.py", line 819, in call_with_http_info 2024-09-29T17:33:10.218Z return self.api_client.call_api( 2024-09-29T17:33:10.219Z ^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.220Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/api_client.py", line 380, in call_api 2024-09-29T17:33:10.221Z return self.__call_api( 2024-09-29T17:33:10.222Z ^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.224Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/api_client.py", line 187, in __call_api 2024-09-29T17:33:10.225Z raise e 2024-09-29T17:33:10.226Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/api_client.py", line 175, in __call_api 2024-09-29T17:33:10.229Z response_data = self.request( 2024-09-29T17:33:10.230Z ^^^^^^^^^^^^^ 2024-09-29T17:33:10.231Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/api_client.py", line 460, in request 2024-09-29T17:33:10.233Z return self.rest_client.POST( 2024-09-29T17:33:10.234Z ^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:33:10.235Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/rest.py", line 345, in POST 2024-09-29T17:33:10.236Z return self.request( 2024-09-29T17:33:10.237Z ^^^^^^^^^^^^^ 2024-09-29T17:33:10.238Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/rest.py", line 279, in request 2024-09-29T17:33:10.239Z raise PineconeApiException(http_resp=r) 2024-09-29T17:33:10.240Z pinecone.core.openapi.shared.exceptions.PineconeApiException: (429) 2024-09-29T17:33:10.241Z Reason: Too Many Requests 2024-09-29T17:33:10.243Z HTTP response headers: HTTPHeaderDict({'Date': 'Sun, 29 Sep 2024 17:33:10 GMT', 'Content-Type': 'application/json', 'Content-Length': '168', 'Connection': 'keep-alive', 'x-pinecone-request-latency-ms': '0', 'x-pinecone-request-id': '4539119554456703513', 'x-envoy-upstream-service-time': '1', 'server': 'envoy'}) 2024-09-29T17:33:10.244Z HTTP response body: {"code":8,"message":"Request failed. You've reached the max query read units per second for index hhspro (2000 read units per second). Pace your queries.","details":[]} 2024-09-29T17:33:10.245Z INFO Exiting actor ({"exit_code": 1})

It is just weird, because suddenly they all fail. They have been working for 6 months every day.

I also tried with the new beta commit.

2024-09-29T17:48:43.129Z ACTOR: Pulling Docker image of build eOAJxfCoUHOpZ5x6h from repository. 2024-09-29T17:48:54.173Z ACTOR: Creating Docker container. 2024-09-29T17:48:54.755Z ACTOR: Starting Docker container. 2024-09-29T17:48:57.989Z INFO Initializing actor... 2024-09-29T17:48:57.991Z INFO System info ({"apify_sdk_version": "1.7.2", "apify_client_version": "1.6.4", "python_version": "3.11.10", "os": "linux"}) 2024-09-29T17:48:57.993Z INFO Starting the Vector Store Actor 2024-09-29T17:48:58.122Z INFO Received start argument (vector database name): pinecone 2024-09-29T17:48:58.125Z INFO Get embeddings class: OpenAI 2024-09-29T17:48:58.933Z INFO Load Dataset ID dBbM1MMyBsdGfGhQx and extract fields ['Product URL', 'Titel', 'Pris', 'Pris_før_rabat', 'Image URL', 'Lagerstatus', 'Tilbud', 'Kort_beskrivelse', 'Yderligere_information', 'Beskrivelse'] 2024-09-29T17:48:59.600Z INFO Dataset loaded, number of documents: 1410 2024-09-29T17:48:59.677Z INFO Documents chunked to 1411 chunks 2024-09-29T17:49:00.481Z INFO Update database with crawled data. Delta updates enabled 2024-09-29T17:49:00.483Z INFO Comparing crawled data with the database ... 2024-09-29T17:49:00.485Z INFO Processing item_id 94499446dafd3b6d12856d79c456c93081c0b426430b33dcbbc61f183d6f23ba (0/1410) to compare crawled data with the database 2024-09-29T17:49:01.929Z ERROR (429) 2024-09-29T17:49:01.932Z Reason: Too Many Requests 2024-09-29T17:49:01.934Z HTTP response headers: HTTPHeaderDict({'Date': 'Sun, 29 Sep 2024 17:49:01 GMT', 'Content-Type': 'application/json', 'Content-Length': '168', 'Connection': 'keep-alive', 'x-pinecone-request-latency-ms': '3', 'x-pinecone-request-id': '3386755732814587719', 'x-envoy-upstream-service-time': '1', 'server': 'envoy'}) 2024-09-29T17:49:01.936Z HTTP response body: {"code":8,"message":"Request failed. You've reached the max query read units per second for index hhspro (2000 read units per second). Pace your queries.","details":[]} 2024-09-29T17:49:01.938Z 2024-09-29T17:49:01.941Z ERROR Actor failed with an exception 2024-09-29T17:49:01.943Z Traceback (most recent call last): 2024-09-29T17:49:01.945Z File "/usr/src/app/src/main.py", line 69, in run_actor 2024-09-29T17:49:01.948Z update_db_with_crawled_data(vcs_, documents) 2024-09-29T17:49:01.950Z File "/usr/src/app/src/vcs.py", line 60, in update_db_with_crawled_data 2024-09-29T17:49:01.952Z data_add, ids_update_last_seen, ids_del = compare_crawled_data_with_db(vector_store, documents) 2024-09-29T17:49:01.955Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:49:01.957Z File "/usr/src/app/src/vcs.py", line 104, in compare_crawled_data_with_db 2024-09-29T17:49:01.959Z crawled_db[item_id] = vector_store.get_by_item_id(item_id) 2024-09-29T17:49:01.961Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:49:01.963Z File "/usr/src/app/src/vector_stores/pinecone.py", line 43, in get_by_item_id 2024-09-29T17:49:01.965Z results = self.index.query( 2024-09-29T17:49:01.967Z ^^^^^^^^^^^^^^^^^ 2024-09-29T17:49:01.969Z File "/usr/local/lib/python3.11/site-packages/pinecone/utils/error_handling.py", line 11, in inner_func 2024-09-29T17:49:01.971Z return func(*args, **kwargs) 2024-09-29T17:49:01.973Z ^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:49:01.977Z File "/usr/local/lib/python3.11/site-packages/pinecone/data/index.py", line 429, in query 2024-09-29T17:49:01.979Z response = self._vector_api.query( 2024-09-29T17:49:01.981Z ^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:49:01.983Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/api_client.py", line 761, in call 2024-09-29T17:49:01.985Z return self.callable(self, *args, **kwargs) 2024-09-29T17:49:01.987Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:49:01.989Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/data/api/data_plane_api.py", line 639, in __query 2024-09-29T17:49:01.991Z return self.call_with_http_info(**kwargs) 2024-09-29T17:49:01.992Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:49:01.994Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/api_client.py", line 819, in call_with_http_info 2024-09-29T17:49:01.996Z return self.api_client.call_api( 2024-09-29T17:49:01.998Z ^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:49:02.000Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/api_client.py", line 380, in call_api 2024-09-29T17:49:02.003Z return self.__call_api( 2024-09-29T17:49:02.005Z ^^^^^^^^^^^^^^^^ 2024-09-29T17:49:02.006Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/api_client.py", line 187, in __call_api 2024-09-29T17:49:02.008Z raise e 2024-09-29T17:49:02.010Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/api_client.py", line 175, in __call_api 2024-09-29T17:49:02.012Z response_data = self.request( 2024-09-29T17:49:02.014Z ^^^^^^^^^^^^^ 2024-09-29T17:49:02.016Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/api_client.py", line 460, in request 2024-09-29T17:49:02.018Z return self.rest_client.POST( 2024-09-29T17:49:02.020Z ^^^^^^^^^^^^^^^^^^^^^^ 2024-09-29T17:49:02.023Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/rest.py", line 345, in POST 2024-09-29T17:49:02.025Z return self.request( 2024-09-29T17:49:02.027Z ^^^^^^^^^^^^^ 2024-09-29T17:49:02.029Z File "/usr/local/lib/python3.11/site-packages/pinecone/core/openapi/shared/rest.py", line 279, in request 2024-09-29T17:49:02.032Z raise PineconeApiException(http_resp=r) 2024-09-29T17:49:02.034Z pinecone.core.openapi.shared.exceptions.PineconeApiException: (429) 2024-09-29T17:49:02.036Z Reason: Too Many Requests 2024-09-29T17:49:02.038Z HTTP response headers: HTTPHeaderDict({'Date': 'Sun, 29 Sep 2024 17:49:01 GMT', 'Content-Type': 'application/json', 'Content-Length': '168', 'Connection': 'keep-alive', 'x-pinecone-request-latency-ms': '3', 'x-pinecone-request-id': '3386755732814587719', 'x-envoy-upstream-service-time': '1', 'server': 'envoy'}) 2024-09-29T17:49:02.041Z HTTP response body: {"code":8,"message":"Request failed. You've reached the max query read units per second for index hhspro (2000 read units per second). Pace your queries.","details":[]} 2024-09-29T17:49:02.043Z INFO Exiting actor ({"exit_code": 1})

so if i Enable incremental updates, it fails, if not, it does not fail.

Yeah, that’s definitely the easiest fix. However, I'm still trying to fully understand what’s happening

From you run: https://console.apify.com/admin/users/Zkd8wkb2TsUSw30xY/actors/runs/wNpsDIuBYerAQeU1i#log

This log is saying that less than 100 request were executed and then we got 429 too many requests.

Can you please check the status of your Pinecone database? I've attached my run for reference.

But it is not really a fix, because I somehow need to delete the old vectors when I upsert again. And the metrics looks fine, this is the run. xhJcKQ3fVsGS8NAZE

But now, if I create a new pinecone index, it seems like the problem is gone, but all my old pinecones does not work, even on different accounts. Will try more.

I'm pleased to see that you're using the incremental update feature, but I'm also disappointed that it's not working for you right now.

At this point, I’m inclined to think that the issue might be related to Pinecone itself. Could you reach out to their support for clarification?

In the meantime, I'll implement a retry mechanism with a delay on errors so that you can test it more.

Yeah, i have reached out to them. Maybe it is pinecone, waiting for answers. But it seems now like if I create a new flow in pinecone, the problem is gone. But all the old flows has the same problem

I was able to reproduce the issue on my end and encountered the same "429 Too Many Requests" error for my index as well.

To address this, I implemented an exponential backoff feature and reran the integration successfully. It’s now published as a beta release.
The downside is that the integration takes a longer time to finish (13 minutes for me). 😕 Please make sure to set the memory limit to 512 MB — that should be sufficient (instead of 1024 MB, not to waste resources)

Could you please give it a try and let me know if it works for you? 🙏

It works now, it does not take much longer for me. 3 minutes compared to 1.5 minutes before. Great, thanks man!