New storage system for Request Queue is out!

We’re excited to announce the launch of Apify's new Request Queue storage system. This update enhances our platform by unlocking new use cases and addressing common issues with previous request queue functionalities.

New use cases

Distributed scraping: The new system integrates a locking mechanism that allows multiple clients to process the same request queue without duplicating work. This feature is crucial for distributed scraping tasks, where multiple Actor runs scrape one request queue.

Batch operations: The new storage system supports batch operations, allowing multiple requests to be enqueued or dequeued in a single operation. This reduces network latency and accelerates the processing of large volumes of requests.

Unlimited data retention: Unlike the previous system where each request had a fixed expiration date, the new system allows for indefinite retention of requests in named queues. This feature facilitates incremental crawling by allowing you to append new URLs to the queue and pick up where you left off in subsequent runs.

For more detailed information on the new Request Queue system, please refer to our blog post. To learn how to implement these features directly, you can explore the Apify documentation or review the tutorials available at Apify Academy.