Actor Benchmark

Benchmarks multiple builds of any Apify actor in parallel and compares them on whatever you choose to measure. By default it reports run duration, dataset item count, total cost, items per second, and cost per item.

Each build is run N times with the same input. You decide what is measured in every run and how the runs of a build are turned into statistics, by providing two JavaScript functions (see Post-processing). The statistics are presented as a table and a set of bar charts, and an AI model can optionally write a report on them.

How it works

1. Dispatch — all runs (N builds × M runs each) are started in parallel as individual actor calls. A failure in one run does not abort the others.
2. Re-fetch — once all runs finish, each run is re-fetched from the API to ensure stats are fully settled.
3. Extract — your extractRunData function is called for every run and returns the values you want to measure for it.
4. Aggregate — for each build, your aggregate function is called once with the extracted values of that build's succeeded runs (see Error handling), and returns the statistics to report (averages, medians, …).
5. Output — the statistics are pushed to the default dataset, the extracted values of every run to the allRuns dataset, and one bar chart per numeric statistic is optionally saved to the key-value store.
6. Report — if you set a reportPrompt, an AI model reads the statistics and writes a report (see AI report).

Modes

The benchmark supports two mutually exclusive modes:

• Actor mode — provide actorId + actorInput. Each run calls the actor with the given input.
• Task mode — provide taskId only. Each run calls the task using its own saved input on Apify.

If both are provided, actor mode takes priority and a warning is logged. If neither is provided, the actor exits with an error.

Input

Field	Type	Required	Description
buildNumbers	string[]	yes	List of build numbers or tags to compare (e.g. ["0.3.67", "canary"])
actorId	string	actor mode	ID or name of the actor to benchmark (e.g. apify/website-content-crawler)
actorInput	object	actor mode	JSON input passed identically to every run
taskId	string	task mode	ID or name of the Apify task to benchmark (e.g. my-username/my-task). The task's own saved input is used.
runsPerBuild	integer	no	Number of runs per build, aggregated together (default: 3, min: 1). All runs are started in parallel on your account — ensure you have enough memory quota available to allow them all to start simultaneously.
maxRunDurationSecs	integer	no	Timeout in seconds for each run. If omitted, the actor's default timeout is used (min: 10, no maximum)
memoryMbytes	integer	no	Memory allocated to each run in MB. If omitted, the actor's default memory is used (min: 512, max: 32768)
extractRunData	string	no	JavaScript function called for every run. Defaults to measuring duration, item count and cost. See Post-processing.
aggregate	string	no	JavaScript function called for every build. Defaults to averaging the values measured by the default extractRunData. Change it together with extractRunData.
generateCharts	boolean	no	Save one bar chart per numeric statistic (default: true)
reportPrompt	string	no	Your request to an AI model that writes a report on the results, e.g. Which build is fastest, and is the cost difference worth it?. Leave empty for no report. See AI report.

Post-processing

What is measured and reported is decided by two functions. Each input is a single JavaScript function expression, and it may be async. The two work together: the default aggregate only understands the values returned by the default extractRunData, so change both or neither.

extractRunData

Called once for every run, after the run has finished, including failed runs. It receives one object:

Argument	Description
run	The run object from the Apify API, re-fetched after the run finished
client	An authenticated Apify API client, for example to read the run's dataset or key-value store
build	The build number or tag the run belongs to
runNumber	1-based index of the run within its build
utils	Helpers: sum(values), mean(values) and median(values), taking an array of numbers, and round(value, decimals)

It must return a plain object, which is stored as is in the allRuns dataset and passed to aggregate. The default:

async ({ run, client, utils }) => {
    const durationSecs = (new Date(run.finishedAt) - new Date(run.startedAt)) / 1000;

    let itemCount = 0;
    for await (const _item of client.dataset(run.defaultDatasetId).listItems()) {
        itemCount++;
    }

    const totalCost = run.usageTotalUsd ?? 0;
    return {
        durationSecs: utils.round(durationSecs, 2),
        itemCount,
        totalCost: utils.round(totalCost, 6),
        itemsPerSecond: utils.round(durationSecs > 0 ? itemCount / durationSecs : 0, 4),
        costPerItem: utils.round(itemCount > 0 ? totalCost / itemCount : 0, 8),
    };
};

aggregate

Called once for every build. It receives one object:

Argument	Description
build	The build number or tag
runs	Array of the objects returned by extractRunData for the build's succeeded runs whose extraction worked
utils	The same helpers as above

It must return a plain object with the statistics to report for the build. build is added to it, and it becomes one item of the default dataset. Every field that is a number for all builds gets its own chart. The default:

({ runs, utils }) => {
    const average = (field, decimals) => utils.round(utils.mean(runs.map((run) => run[field])), decimals);

    return {
        durationSecs: average('durationSecs', 2),
        itemCount: average('itemCount', 2),
        totalCost: average('totalCost', 6),
        itemsPerSecond: average('itemsPerSecond', 4),
        costPerItem: average('costPerItem', 8),
    };
};

Each build is aggregated on its own, so aggregate cannot compare builds with each other, for example against a baseline build.

Example

Measuring how many items have no text:

// extractRunData
async ({ run, client }) => {
    let itemCount = 0;
    let emptyTextItems = 0;
    for await (const item of client.dataset(run.defaultDatasetId).listItems()) {
        itemCount++;
        if (!item.text) emptyTextItems++;
    }
    return { itemCount, emptyTextItems };
};

// aggregate
({ runs, utils }) => ({
    medianItemCount: utils.median(runs.map((run) => run.itemCount)),
    emptyTextRate: utils.sum(runs.map((run) => run.emptyTextItems)) / utils.sum(runs.map((run) => run.itemCount)),
});

AI report

Set reportPrompt to have an AI model read the results and write a report on them. The prompt is your request, for example Generate a summary of the benchmark results. or Which build is fastest, and is the cost difference worth it?. If you use custom functions, say what your fields mean, for example that a lower emptyTextRate is better, because the model only sees the field names and values. Leave the prompt empty and no report is generated and nothing is spent.

What is sent to the model:

• the statistics of every build, exactly as aggregate returned them
• whether an actor or a task was benchmarked, its ID, and the number of runs per build

Raw run data (the allRuns dataset) is not sent, and the statistics are marked as untrusted data in the request. The report is generated after the datasets and charts are saved, so it cannot lose any of them. It is saved as report.md in the key-value store and also written to the log.

The request goes through the Apify OpenRouter Actor with the run's token, which starts an additional Actor run and bills the model tokens to your account. A warning is logged at startup when a report is requested, and the tokens used are logged after the request. The model defaults to anthropic/claude-haiku-4.5; it is set by the hidden reportModel input, which is not shown in the Console form but can be set through the API.

Output

Default dataset — statistics per build

One item per build: whatever aggregate returned, plus build and runsSummary. With the default functions:

Field	Description
build	Build tag
runsSummary	Run statuses of the build, e.g. 2/3 SUCCEEDED, 1/3 FAILED
durationSecs	Average wall-clock duration (seconds)
itemCount	Average number of items in the output dataset
totalCost	Average total run cost (USD)
itemsPerSecond	Average throughput (items / second)
costPerItem	Average cost per output item (USD)

Builds without any usable run are left out. A build whose aggregate threw is reported as an item with build, runsSummary and an error message instead, and is not charted.

allRuns dataset — extracted data per run

One item per individual run, including failed runs: whatever extractRunData returned, plus:

Field	Description
build	Build tag
runNumber	1-based index of the run within its build
runId	Apify run ID
status	Final status of the run
error	Only if extractRunData threw for this run: the error message

Key-value store — charts

Unless generateCharts is false, one PNG bar chart is saved for every field of the default dataset that is a number for all builds, with the builds on the X axis and the field's value on the Y axis. The key is chart-<field>, titled <field> per Build. Characters not allowed in key-value store keys are replaced by -, and a numeric suffix is added if two keys would collide. With the default functions:

Key	Field
chart-durationSecs	durationSecs
chart-itemCount	itemCount
chart-totalCost	totalCost
chart-itemsPerSecond	itemsPerSecond
chart-costPerItem	costPerItem

Key-value store — report

If a report was requested and generated, it is saved as report.md (text/markdown).

Error handling

• Empty or invalid function — if extractRunData or aggregate is empty, cannot be parsed, or is not a single function expression, the actor fails before starting any run.
• Single run fails to start — logged as a warning with the error from the orchestrator; remaining runs continue normally.
• Failed runs — a run is succeeded if it finishes with status SUCCEEDED or TIMED-OUT, and failed with any other status. A failed run is logged as a warning with the run's status, exit code, and status message; it is not passed to aggregate but still appears in the allRuns dataset.
• extractRunData throws for a run — logged as a warning; the run appears in the allRuns dataset with an error field and is not passed to aggregate.
• aggregate throws or does not return a plain object for a build — logged as a warning; the build appears in the default dataset with an error field and the other builds are unaffected. The actor exits with a non-zero exit code only if aggregate fails for every build.
• All runs fail, or no run has extracted data — the actor exits with a non-zero exit code and no output is produced.
• Report cannot be generated — if APIFY_TOKEN is not set (for example in a local run), the OpenRouter request fails, times out, or the model returns nothing, a warning is logged and no report is saved. The benchmark itself still succeeds with exit code 0.
• Report requested without a model — if reportPrompt is set but reportModel is empty, the actor fails before starting any run.