Hi Don,

This was my bad as I was releasing a new change.

I've been keeping an eye on that large run of yours and have noticed that its really slowed down. Actually when I try to do new runs myself, the DOT server responds very quickly, hence, I think we are getting explicitly throttled.

I added support for specifying a custom named request queue in the input. For that large run of yours, if you convert its request queue into a "named request queue", and then provide that to any new run of the crawler, then it will just fetch the pages in that queue. Hence, avoiding potential double charges.

Can you give it a try?

Go to the request queue used for that large run: https://console.apify.com/storage/request-queues/e5YS84ON9Ac1Ft6aN

Then under "Actions", click "Rename". Give it a title Then copy the "unique name" value. Paste that "unique name" as the "Request Queue Name" in the actor input options.

Also try specifying residential proxies.

Here's the latest output, resulted in a failure.

2025-05-03T14:40:20.862Z ACTOR: Pulling Docker image of build CPhOneZD58G4Z7FkB from registry.

2025-05-03T14:40:23.946Z ACTOR: Creating Docker container.

2025-05-03T14:40:24.118Z ACTOR: Starting Docker container.

2025-05-03T14:40:26.095Z INFO CrawlGuard initialized with actorId: MB6MTbLZh4tkXpJxg and runId: VeUyt3OeF8xCVW97L

2025-05-03T14:40:26.100Z INFO System info {"apifyVersion":"3.2.6","apifyClientVersion":"2.11.0","crawleeVersion":"3.12.1","osType":"Linux","nodeVersion":"v18.20.8"}

2025-05-03T14:40:26.112Z INFO Loading input

2025-05-03T14:40:26.271Z INFO input: {

2025-05-03T14:40:26.274Z "dot_start": 1412712,

2025-05-03T14:40:26.276Z "filter_mode": "DOT_RANGE",

2025-05-03T14:40:26.278Z "is_premium_mode": true,

2025-05-03T14:40:26.280Z "max_requests_per_crawl": 3000000,

2025-05-03T14:40:26.282Z "proxy_configuration": {

2025-05-03T14:40:26.284Z "useApifyProxy": true,

2025-05-03T14:40:26.286Z "apifyProxyGroups": [

2025-05-03T14:40:26.288Z "RESIDENTIAL"

2025-05-03T14:40:26.290Z ],

2025-05-03T14:40:26.292Z "apifyProxyCountry": "US"

2025-05-03T14:40:26.294Z },

2025-05-03T14:40:26.296Z "request_queue_name": "Large test run"

2025-05-03T14:40:26.298Z }

2025-05-03T14:40:26.489Z INFO Initializing DotLookup for tag: 2025Apr

2025-05-03T14:40:26.491Z lookups initialized

2025-05-03T14:40:29.043Z node:internal/process/esm_loader:40

2025-05-03T14:40:29.049Z internalBinding('errors').triggerUncaughtException(

2025-05-03T14:40:29.051Z ^

2025-05-03T14:40:29.053Z

2025-05-03T14:40:29.055Z ApifyApiError: Invalid value provided in queue: name can only contain letters 'a' through 'z', the digits '0' through '9', and the hyphen ('-') but only in the middle of the string (e.g. 'my-value-1')

2025-05-03T14:40:29.060Z clientMethod: RequestQueueCollectionClient.getOrCreate

2025-05-03T14:40:29.062Z statusCode: 400

2025-05-03T14:40:29.064Z type: schema-validation

2025-05-03T14:40:29.067Z attempt: 1

2025-05-03T14:40:29.070Z httpMethod: post

2025-05-03T14:40:29.073Z path: /v2/request-queues

2025-05-03T14:40:29.076Z stack:

2025-05-03T14:40:29.081Z at makeRequest (/usr/src/app/ node_modules/ apify-client /dist/http_client.js:187:30 )

2025-05-03T14:40:29.083Z at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

2025-05-03T14:40:29.085Z at async RequestQueueCollectionClient._getOrCreate (/usr/src/app/ node_modules/ apify-client /dist/base/resource_collection_client.js:32:26 )

2025-05-03T14:40:29.088Z at async StorageManager.openStorage (/usr/src/app/ node_modules/ @crawlee /core/storages/storage_manager.js:81:35 )

2025-05-03T14:40:29.090Z at async BufferedRequestQueueManager.init (file:///usr/src/app/ src/shared-libs/BufferedRequestQueueManager.mjs:23:29 )

2025-05-03T14:40:29.096Z at async main (file:///usr/src/app/ src/main.js:130:5 )

2025-05-03T14:40:29.098Z at async file:///usr/src/app/ src/main.js:410:1

2025-05-03T14:40:29.099Z data: undefined {

2025-05-03T14:40:29.102Z clientMethod: 'RequestQueueCollectionClient.getOrCreate' ,

2025-05-03T14:40:29.105Z statusCode: 400 ,

2025-05-03T14:40:29.107Z type: 'schema-validation' ,

2025-05-03T14:40:29.110Z attempt: 1 ,

2025-05-03T14:40:29.112Z httpMethod: 'post' ,

2025-05-03T14:40:29.115Z path: '/v2/request-queues' ,

2025-05-03T14:40:29.117Z originalStack: '\n' +

2025-05-03T14:40:29.119Z ' at makeRequest (/usr/src/app/node_modules/apify-client/dist/http_client.js:187:30)\n' +

2025-05-03T14:40:29.121Z ' at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n' +

2025-05-03T14:40:29.123Z ' at async RequestQueueCollectionClient._getOrCreate (/usr/src/app/node_modules/apify-client/dist/base/resource_collection_client.js:32:26)\n' +

2025-05-03T14:40:29.125Z ' at async StorageManager.openStorage (/usr/src/app/node_modules/@crawlee/core/storages/storage_manager.js:81:35)\n' +

2025-05-03T14:40:29.127Z ' at async BufferedRequestQueueManager.init (file:///usr/src/app/src/shared-libs/BufferedRequestQueueManager.mjs:23:29)\n' +

2025-05-03T14:40:29.129Z ' at async main (file:///usr/src/app/src/main.js:130:5)\n' +

2025-05-03T14:40:29.131Z ' at async file:///usr/src/app/src/main.js:410:1' ,

2025-05-03T14:40:29.133Z data: undefined

2025-05-03T14:40:29.135Z }

2025-05-03T14:40:29.137Z

2025-05-03T14:40:29.139Z Node.js v18.20.8

This is Don Richards-Boeff's card. Their email is don.richardsboeff@roadsync.com. Their phone number is +1 404 398 5527. ( https://hihello.me/p/8c1974f7-8c34-49da-bce4-600c1871fdcd )

Confirming on the new run, residential, and the named queue with no spaces, seeing about 1K records every 5 min. Will continue monitoring. Not using the old run any longer, just starting with the approximate end point and proceeding. At the point where I need the records more than a few missed here and there.

I take that back, throttling does start to kick in. Wondering if removing the location (i.e. United States) and leaving blank will allow it to rotate residential proxies to multiple locations and see better performance. At the moment, I see roughly similar behavior from Data center and Home after 500+ records (very fast) and subsequent (starts to slow).

1. I don't see the request queue getting renamed on my side. https://console.apify.com/storage/request-queues/e5YS84ON9Ac1Ft6aN Also, you need to pass in the full "unique name". It will be your username/title. Currently I see the run not picking up the named request queue.

2. The throttling is a problem. I will need some time here to investigate this more and come up with an alternative solution. Will keep you posted.

All good.

I'm unclear on the steps to take re: named queue. On the newest running actor, I gave it a queue name "largetestrun".

Re: throttling, may be something APIfy can assist with feedback on, too. Seems like a multiple / rotating proxy need. Sending batches from one proxy, moving to another, and so on, so requests aren't always coming from the same location / IP. Might need user agent rotation. If you see a pattern, let me know, can see if our team has other ideas to offer.

This is Don Richards-Boeff's card. Their email is don.richardsboeff@roadsync.com. Their phone number is +1 404 398 5527. ( https://hihello.me/p/8c1974f7-8c34-49da-bce4-600c1871fdcd )