# Actor Benchmark (`apify/actor-benchmark`) Actor Compares various builds of the same actor to measure how they perform on the same input - **URL**: https://apify.com/apify/actor-benchmark.md - **Developed by:** [Apify](https://apify.com/apify) (Apify) - **Categories:** Automation, Developer tools - **Stats:** 2 total users, 0 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Actor Benchmark Benchmarks multiple builds of any Apify actor in parallel and compares their performance across key metrics: run duration, dataset item count, total cost, items per second, and cost per item. Each build is run N times with the same input. Results are averaged per build to reduce the effect of outliers, then presented as a table and a set of bar charts. ### How it works 1. **Dispatch** — all runs (N builds × M runs each) are started in parallel as individual actor calls. A failure in one run does not abort the others. 2. **Re-fetch** — once all runs finish, each run is re-fetched from the API to ensure stats are fully settled. 3. **Item count** — the real item count is read from the Dataset API for each run's default dataset. 4. **Metrics** — per-run metrics are computed and averaged per build. Failed runs are excluded from averages but still recorded for transparency. 5. **Output** — averaged results are pushed to the default dataset, raw per-run rows to the `allRuns` dataset, and five bar charts are saved to the key-value store. ### Modes The benchmark supports two mutually exclusive modes: - **Actor mode** — provide `actorId` + `actorInput`. Each run calls the actor with the given input. - **Task mode** — provide `taskId` only. Each run calls the task using its own saved input on Apify. If both are provided, actor mode takes priority and a warning is logged. If neither is provided, the actor exits with an error. ### Input | Field | Type | Required | Description | | -------------------- | -------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `buildNumbers` | string[] | yes | List of build numbers or tags to compare (e.g. `["0.3.67", "canary"]`) | | `actorId` | string | actor mode | ID or name of the actor to benchmark (e.g. `apify/website-content-crawler`) | | `actorInput` | object | actor mode | JSON input passed identically to every run | | `taskId` | string | task mode | ID or name of the Apify task to benchmark (e.g. `my-username/my-task`). The task's own saved input is used. | | `runsPerBuild` | integer | no | Number of runs per build, averaged together (default: `3`, min: `1`). All runs are started in parallel on your account — ensure you have enough memory quota available to allow them all to start simultaneously. | | `maxRunDurationSecs` | integer | no | Timeout in seconds for each run. If omitted, the actor's default timeout is used (min: `10`, max: `36000`) | | `memoryMbytes` | integer | no | Memory allocated to each run in MB. If omitted, the actor's default memory is used (min: `512`, max: `32768`) | ### Output #### Default dataset — averaged results One row per build, with all numeric fields averaged across the successful runs of that build. | Field | Description | | ---------------- | --------------------------------------------- | | `build` | Build tag | | `status` | e.g. `3/3 SUCCEEDED` or `2/3 SUCCEEDED` | | `durationSecs` | Average wall-clock duration (seconds) | | `itemCount` | Average number of items in the output dataset | | `totalCost` | Average total run cost (USD) | | `itemsPerSecond` | Average throughput (items / second) | | `costPerItem` | Average cost per output item (USD) | #### `allRuns` dataset — raw per-run data One row per individual run, including failed runs. Same fields as above plus: | Field | Description | | ----------- | ----------------------------------------- | | `runNumber` | 1-based index of the run within its build | | `runId` | Apify run ID | #### Key-value store — charts Five PNG bar charts comparing builds side by side, saved under these keys: | Key | Metric | | ------------------------ | --------------------------- | | `chart-duration` | Run Duration per Build (s) | | `chart-items` | Dataset Items per Build | | `chart-total-cost` | Total Cost per Build ($) | | `chart-items-per-second` | Items per Second per Build | | `chart-cost-per-item` | Cost per Item per Build ($) | ### Error handling - **Single run fails to start** — logged as a warning with the error from the orchestrator; remaining runs continue normally. - **Run finishes with non-SUCCEEDED status** — logged as a warning with the run's status, exit code, and status message; the run is excluded from averaged metrics but still appears in the `allRuns` dataset. - **All runs fail** — the actor exits with a non-zero exit code and no output is produced. # Actor input Schema ## `buildNumbers` (type: `array`): List of build numbers or tags to benchmark. Each entry will receive an identical run in parallel. You can use exact version numbers (e.g. "0.3.67") or tags (e.g. "canary"), which will use the latest build associated with that tag. ## `runsPerBuild` (type: `integer`): Number of times each build is run. Results are averaged to reduce the effect of outliers. All runs are started in parallel on your account, so make sure you have enough memory quota available to allow them all to start simultaneously. ## `actorId` (type: `string`): The ID or name of the Apify actor to benchmark (e.g. "apify/website-content-crawler"). ## `actorInput` (type: `object`): The JSON input passed to every run of the benchmarked actor. ## `taskId` (type: `string`): The ID or name of an Apify task to benchmark (e.g. "my-username/my-task"). ## `maxRunDurationSecs` (type: `integer`): Timeout in seconds for each actor run. Runs exceeding this limit will be aborted. ## `memoryMbytes` (type: `integer`): Memory in megabytes allocated to each actor run. ## Actor input object example ```json { "buildNumbers": [ "0.3.67", "canary" ], "runsPerBuild": 3, "actorId": "apify/website-content-crawler", "actorInput": { "maxResults": 1, "blockMedia": true, "clickElementsCssSelector": "[aria-expanded=\"false\"]", "clientSideMinChangePercentage": 15, "crawlerType": "playwright:adaptive", "proxyConfiguration": { "useApifyProxy": true }, "readableTextCharThreshold": 100, "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, img[src^='data:'],\n[role=\"alert\"],\n[role=\"banner\"],\n[role=\"dialog\"],\n[role=\"alertdialog\"],\n[role=\"region\"][aria-label*=\"skip\" i],\n[aria-modal=\"true\"]", "renderingTypeDetectionPercentage": 10, "respectRobotsTxtFile": true, "reuseStoredDetectionResults": false, "signHttpRequests": false, "startUrls": [ { "url": "https://docs.apify.com/academy/scraping-basics-javascript" } ], "storeSkippedUrls": false, "useLlmsTxt": false, "useSitemaps": false, "keepUrlFragments": false, "ignoreCanonicalUrl": false, "ignoreHttpsErrors": false, "removeCookieWarnings": true, "expandIframes": true, "aggressivePrune": false, "debugMode": false, "debugLog": false, "saveHtml": false, "saveHtmlAsFile": false, "saveMarkdown": true, "saveFiles": false, "saveScreenshots": false } } ```` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "buildNumbers": [ "0.3.67", "canary" ], "actorId": "apify/website-content-crawler", "actorInput": { "maxResults": 1, "blockMedia": true, "clickElementsCssSelector": "[aria-expanded=\"false\"]", "clientSideMinChangePercentage": 15, "crawlerType": "playwright:adaptive", "proxyConfiguration": { "useApifyProxy": true }, "readableTextCharThreshold": 100, "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, img[src^='data:'],\n[role=\"alert\"],\n[role=\"banner\"],\n[role=\"dialog\"],\n[role=\"alertdialog\"],\n[role=\"region\"][aria-label*=\"skip\" i],\n[aria-modal=\"true\"]", "renderingTypeDetectionPercentage": 10, "respectRobotsTxtFile": true, "reuseStoredDetectionResults": false, "signHttpRequests": false, "startUrls": [ { "url": "https://docs.apify.com/academy/scraping-basics-javascript" } ], "storeSkippedUrls": false, "useLlmsTxt": false, "useSitemaps": false, "keepUrlFragments": false, "ignoreCanonicalUrl": false, "ignoreHttpsErrors": false, "removeCookieWarnings": true, "expandIframes": true, "aggressivePrune": false, "debugMode": false, "debugLog": false, "saveHtml": false, "saveHtmlAsFile": false, "saveMarkdown": true, "saveFiles": false, "saveScreenshots": false } }; // Run the Actor and wait for it to finish const run = await client.actor("apify/actor-benchmark").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "buildNumbers": [ "0.3.67", "canary", ], "actorId": "apify/website-content-crawler", "actorInput": { "maxResults": 1, "blockMedia": True, "clickElementsCssSelector": "[aria-expanded=\"false\"]", "clientSideMinChangePercentage": 15, "crawlerType": "playwright:adaptive", "proxyConfiguration": { "useApifyProxy": True }, "readableTextCharThreshold": 100, "removeElementsCssSelector": """nav, footer, script, style, noscript, svg, img[src^='data:'], [role=\"alert\"], [role=\"banner\"], [role=\"dialog\"], [role=\"alertdialog\"], [role=\"region\"][aria-label*=\"skip\" i], [aria-modal=\"true\"]""", "renderingTypeDetectionPercentage": 10, "respectRobotsTxtFile": True, "reuseStoredDetectionResults": False, "signHttpRequests": False, "startUrls": [{ "url": "https://docs.apify.com/academy/scraping-basics-javascript" }], "storeSkippedUrls": False, "useLlmsTxt": False, "useSitemaps": False, "keepUrlFragments": False, "ignoreCanonicalUrl": False, "ignoreHttpsErrors": False, "removeCookieWarnings": True, "expandIframes": True, "aggressivePrune": False, "debugMode": False, "debugLog": False, "saveHtml": False, "saveHtmlAsFile": False, "saveMarkdown": True, "saveFiles": False, "saveScreenshots": False, }, } # Run the Actor and wait for it to finish run = client.actor("apify/actor-benchmark").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "buildNumbers": [ "0.3.67", "canary" ], "actorId": "apify/website-content-crawler", "actorInput": { "maxResults": 1, "blockMedia": true, "clickElementsCssSelector": "[aria-expanded=\\"false\\"]", "clientSideMinChangePercentage": 15, "crawlerType": "playwright:adaptive", "proxyConfiguration": { "useApifyProxy": true }, "readableTextCharThreshold": 100, "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, img[src^='\''data:'\''],\\n[role=\\"alert\\"],\\n[role=\\"banner\\"],\\n[role=\\"dialog\\"],\\n[role=\\"alertdialog\\"],\\n[role=\\"region\\"][aria-label*=\\"skip\\" i],\\n[aria-modal=\\"true\\"]", "renderingTypeDetectionPercentage": 10, "respectRobotsTxtFile": true, "reuseStoredDetectionResults": false, "signHttpRequests": false, "startUrls": [ { "url": "https://docs.apify.com/academy/scraping-basics-javascript" } ], "storeSkippedUrls": false, "useLlmsTxt": false, "useSitemaps": false, "keepUrlFragments": false, "ignoreCanonicalUrl": false, "ignoreHttpsErrors": false, "removeCookieWarnings": true, "expandIframes": true, "aggressivePrune": false, "debugMode": false, "debugLog": false, "saveHtml": false, "saveHtmlAsFile": false, "saveMarkdown": true, "saveFiles": false, "saveScreenshots": false } }' | apify call apify/actor-benchmark --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=apify/actor-benchmark", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Actor Benchmark", "description": "Compares various builds of the same actor to measure how they perform on the same input", "version": "0.0", "x-build-id": "H3JLLTGgo8jtRsRv8" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/apify~actor-benchmark/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-apify-actor-benchmark", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/apify~actor-benchmark/runs": { "post": { "operationId": "runs-sync-apify-actor-benchmark", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/apify~actor-benchmark/run-sync": { "post": { "operationId": "run-sync-apify-actor-benchmark", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "buildNumbers" ], "properties": { "buildNumbers": { "title": "Build numbers or tags", "type": "array", "description": "List of build numbers or tags to benchmark. Each entry will receive an identical run in parallel. You can use exact version numbers (e.g. \"0.3.67\") or tags (e.g. \"canary\"), which will use the latest build associated with that tag.", "items": { "type": "string" } }, "runsPerBuild": { "title": "Runs per build", "minimum": 1, "type": "integer", "description": "Number of times each build is run. Results are averaged to reduce the effect of outliers. All runs are started in parallel on your account, so make sure you have enough memory quota available to allow them all to start simultaneously.", "default": 3 }, "actorId": { "title": "Actor ID", "type": "string", "description": "The ID or name of the Apify actor to benchmark (e.g. \"apify/website-content-crawler\")." }, "actorInput": { "title": "Actor input", "type": "object", "description": "The JSON input passed to every run of the benchmarked actor." }, "taskId": { "title": "Task ID", "type": "string", "description": "The ID or name of an Apify task to benchmark (e.g. \"my-username/my-task\")." }, "maxRunDurationSecs": { "title": "Max run duration (seconds)", "minimum": 10, "maximum": 36000, "type": "integer", "description": "Timeout in seconds for each actor run. Runs exceeding this limit will be aborted." }, "memoryMbytes": { "title": "Memory (MB)", "minimum": 512, "maximum": 32768, "type": "integer", "description": "Memory in megabytes allocated to each actor run." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```