Changelog

We’re excited to announce the launch of Apify's new Request Queue storage system. This update enhances our platform by unlocking new use cases and addressing common issues with previous request queue functionalities.

New use cases

Distributed scraping: The new system integrates a locking mechanism that allows multiple clients to process the same request queue without duplicating work. This feature is crucial for distributed scraping tasks, where multiple Actor runs scrape one request queue.

Batch operations: The new storage system supports batch operations, allowing multiple requests to be enqueued or dequeued in a single operation. This reduces network latency and accelerates the processing of large volumes of requests.

Unlimited data retention: Unlike the previous system where each request had a fixed expiration date, the new system allows for indefinite retention of requests in named queues. This feature facilitates incremental crawling by allowing you to append new URLs to the queue and pick up where you left off in subsequent runs.

For more detailed information on the new Request Queue system, please refer to our blog post. To learn how to implement these features directly, you can explore the Apify documentation or review the tutorials available at Apify Academy.

Our redesigned Store is now finally accessible on our website! We introduce new design, sections, Actor collections, and featured developers. All in favour to our community and highlighting the amazing Actors you create!

If you're an Actor developer, there are a few things that can help you get into Actor collections or featured developers:

• Good quality README
• Short response time with your Actor issues
• Engaging & supporting Discord community
• Activities promoting your Actors through social media

Join the community and help to grow our Apify Store even faster!

Performance improvements

As part of our continuous performance improvement initiative, we're happy to announce that we successfully improved the Apify API response time by 50% on average and the 90th-percentile startup time of Actors by about 20%. We will continue improving Apify in this direction.

API updates

User limits endpoint now returns maxConcurrentActorJobs and activeActorJobCount properties enabling users to keep an eye on the concurrency limit.

We also added the missing endpoint /actor-builds/:build-id/log, allowing you to quickly access the log of certain builds without a need for an Actor run ID.

Adaptive Playwright Crawler

Try out Crawlee's new AdaptivePlaywrightCrawler class abstraction, which is an extension of PlaywrightCrawler that uses a more limited request handler interface so that it's able to switch to HTTP-only crawling when it detects that it may be possible. This way, you can achieve lower costs when crawling multiple websites.

1const crawler = new AdaptivePlaywrightCrawler({
2    renderingTypeDetectionRatio: 0.1,
3    async requestHandler({ querySelector, pushData, enqueueLinks, request, log }) {
4        // This function is called to extract data from a single web page
5        const $prices = await querySelector('span.price')
6
7        await pushData({
8            url: request.url,
9            price: $prices.filter(':contains("$")').first().text(),
10        })
11
12        await enqueueLinks({ selector: '.pagination a' })
13    },
14});
15
16await crawler.run([
17    'http://www.example.com/page-1',
18    'http://www.example.com/page-2',
19]);

The Apify platform offers a powerful permission system that allows you to easily and securely share your Actors and other resources.

In a recent update we removed the dedicated Delete permission, relying instead on the existing Write permission to guard resource deletion. For example, if you give someone write access to your Actor, they will now be able to delete it.

This should not have any practical impact as already anyone with the Write permission has enough power to break or even erase any resource by overwriting its contents. However, as a precaution, we have already reached out to all our customers who might be affected by this change with an early heads-up.

If you have concerns that your Actors (or other resources) might get accidentally or maliciously deleted, be assured that you can always reach out to our support. For a limited amount of time we can recover the Actor for you. Also, a good practice is to use Git for more complex Actors, which makes recovering older Actor versions very easy.

This change allows us to simplify the overall system and bring more exciting features to the Apify platform.

List of resources that will no longer offer a dedicated Delete permission:

• Actor
• Actor task
• Key-value store
• Dataset
• Request queue
• Schedule

Get ready for an even better Apify Store!

We're making changes to make our Store more flexible and prepared for future growth! These changes are critical for our community developers and Actor users. Be prepared! We're starting slow but you shouldn't miss it.

New developer profiles

Our goal is to highlight what our community developers can bring to the table. We've updated their profiles, included more social media links, and added a new section for them to introduce their skills in more depth.

Check out your profile settings in Apify Console and make sure your profile shines!

Better communication with your users

We launched an email functionality which you can use to inform your Actor users about the most important updates. This can be done once a month to reduce spam and ensures everybody is up to date!

Store changes in Console

We redesigned how our Store looks in Apify Console and introduced new sections, where we want to highlight the most interesting Actors. The goal is to progressively push community Actors to the front and make sure your work is visible!

Actor collections in Console

We've prepared specific collections of Actors and have taken the time to pick the best within each category. We'll start with:

• Apify picks
• Social media essentials
• Actors from community experts

Some collections will be quite static, while others will be updated regularly. But this is only the first step! Once we're out on the web, we'll get you more involved and things will start to be even more interesting!