# Custom Run Async Endpoint (`sovanza.inc/custom-run-async-endpoint`) Actor Custom Run Async Endpoint runs an async HTTP API inside an Apify Actor container. Submit jobs, poll status, wait for results, control concurrency, protect routes with Bearer auth, and save completed job outputs to the dataset. - **URL**: https://apify.com/sovanza.inc/custom-run-async-endpoint.md - **Developed by:** [Sovanza](https://apify.com/sovanza.inc) (community) - **Categories:** Developer tools, Automation, Integrations - **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing from $5.00 / 1,000 completed jobs This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ### Async API Runner – Custom Endpoint Execution & Background Processing Run **asynchronous workloads** behind a **hosted HTTP API** on your Actor’s [container web server](https://docs.apify.com/platform/actors/development/programming-interface/container-web-server) URL. Submit jobs, poll for status, or **wait** longer than typical synchronous proxies allow — ideal when you need **queued execution**, **controlled concurrency**, and **structured results** in the Apify dataset without building extra infrastructure. ### What is Custom Run Async Endpoint and How Does It Work? Custom Run Async Endpoint is an Apify Actor that runs a long-lived **Express** HTTP server inside your Actor container. Clients call the **Container URL** to enqueue work; an internal worker pool executes tasks with a configurable concurrency cap; each finished job is written as **one dataset row** for audit and downstream automation. This Actor is designed for: - Developers building async handoffs on Apify - Automation engineers who need **202 Accepted** job submission - Teams that want **dataset-auditable** task outcomes - Integration prototypes with **echo**, **sleep**, or guarded **fetch** workloads **What this repo is:** `src/main.js` implements job enqueue, an internal queue (`maxConcurrentTasks`), optional **Bearer** auth, blocking wait endpoints, and dataset export per job. **What it is not:** a universal “retry any REST method with arbitrary JSON/form bodies” integration engine. Tasks are **`echo`**, **`sleep`**, or (optionally) **`fetch`** wrappers — extend the codebase or call from your own service for full REST orchestration. ### Why Use This Async API Runner? Use this Actor when: - Upstream gateways time out before your work finishes - You need **async submission** with optional **blocking wait** on the same run - You want to cap parallel work with **`maxConcurrentTasks`** - You need a **dataset trail** of every completed or failed job - You already run on Apify and want a container URL without separate hosting ➡️ Best for controlled, auditable background-style processing inside an Apify run — not a detached serverless fleet. ### Core Capabilities | Capability | Description | |------------|-------------| | **Async submission** | `POST /tasks` returns `jobId` immediately (`202`); work continues if the client disconnects | | **Concurrency pool** | `maxConcurrentTasks` (1–50) limits parallel workers | | **Blocking waits** | `POST /tasks/:jobId/wait` and `POST /run-and-wait` poll until terminal status or timeout | | **Optional Bearer auth** | `accessToken` in input → protected routes require `Authorization: Bearer …` or `?token=` (except `/health`) | | **Guarded fetch** | Outbound HTTP `fetch` tasks only when `allowFetchTasks` is `true` (off by default) | | **Dataset audit** | Each terminal job pushes one row with status, task, result/error, timestamps | **Retries:** automatic HTTP retry/backoff is **not** implemented in `main.js`. Add retry logic in clients calling this API or extend the worker. ### Execution Flow 1. Enable **Container web server** in Actor settings; Apify assigns **`ACTOR_WEB_SERVER_PORT`** and a public **Container URL**. 2. HTTP server listens on that port (default **4321** locally). 3. Client **`POST /tasks`** with task JSON → job **`pending`** → worker **`running`** → **`completed`** or **`failed`**. 4. Terminal jobs call **`Actor.pushData`** with `jobId`, `task`, `result`/`error`, timestamps. 5. Wait endpoints poll every ~200 ms until done or **timeout** (clamped by input). ### HTTP Routes | Method | Path | Description | |--------|------|-------------| | **GET** | `/health` | Liveness `{ ok, service, activeTasks, queued }` — **no auth** | | **GET** | `/` | Discovery JSON (routes + optional `containerUrl`) | | **POST** | `/tasks` | Body = task JSON → **202** `{ jobId, getUrl, waitUrl }` | | **GET** | `/tasks/:jobId` | Full job snapshot | | **POST** | `/tasks/:jobId/wait` | Block until terminal or timeout (`timeoutMs` query/body) | | **POST** | `/run-and-wait` | Enqueue task + block until terminal or timeout | ### Task JSON (Supported Workloads) Default type is **`echo`** if omitted. | `type` | Example body | Behaviour | |--------|----------------|-----------| | **`echo`** | `{ "type": "echo", "payload": { … } }` | Returns structured echo result (safe demos) | | **`sleep`** | `{ "type": "sleep", "ms": 5000 }` | Async delay capped by **`maxSleepMs`** | | **`fetch`** | `{ "type": "fetch", "url": "https://…", "method": "GET", "headers": { } }` | Outbound **`fetch()`** only if **`allowFetchTasks`** is **`true`**. **SSRF risk** — use **`accessToken`** and trusted callers only. No configurable request **body** in shipped code. | ### How to Use on Apify #### Using the Actor 1. **Go to Custom Run Async Endpoint** on the Apify platform. 2. **Enable Container web server** in Actor **Settings** → **Container web server**. 3. **Configure input** (optional `accessToken`, concurrency, wait timeouts, `allowFetchTasks`). 4. **Start a run** and copy the **Container URL** from the run page. 5. Call **`GET /health`**, then **`POST /tasks`** with your task JSON. 6. Poll **`GET /tasks/:jobId`**, block with **`POST /tasks/:jobId/wait`**, or use **`POST /run-and-wait`**. 7. Open the **Dataset** tab (or Output schema links) for finished job rows. #### Input Configuration ```json { "accessToken": "your-secret-token", "maxConcurrentTasks": 5, "defaultWaitTimeoutMs": 600000, "maxWaitTimeoutMs": 3600000, "allowFetchTasks": false, "maxSleepMs": 3600000 } ```` | Field | Description | |--------|-------------| | `accessToken` | Optional shared secret (**secret** input). Protects all routes except `GET /health`. | | `maxConcurrentTasks` | Max parallel workers (default `5`, range 1–50). | | `defaultWaitTimeoutMs` | Default wait window when client omits `timeoutMs` (default `600000`). | | `maxWaitTimeoutMs` | Hard cap for any wait timeout (default `3600000`). | | `allowFetchTasks` | Must be **`true`** to enable outbound **`fetch`** tasks (default `false`). | | `maxSleepMs` | Maximum milliseconds for **`sleep`** tasks (default `3600000`). | ### Output The Actor exposes two output surfaces (see **Output** schema in Console): | Output | Description | |--------|-------------| | **Container API** | Live base URL (`{{run.containerUrl}}`) while the run is active — use for `/health`, `/tasks`, wait routes | | **Job results** | Default dataset — one row per **completed** or **failed** job | #### Dataset fields (per job) | Field | Description | |--------|-------------| | `jobId` | UUID assigned at enqueue | | `status` | `completed` or `failed` | | `task` | Original task JSON | | `result` | Structured result object when successful | | `error` | Error message when failed | | `createdAt` | When the job was enqueued | | `startedAt` | When execution started | | `finishedAt` | When the job finished | Example dataset row (echo task): ```json { "jobId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "status": "completed", "task": { "type": "echo", "payload": { "message": "hello" } }, "result": { "taskType": "echo", "payload": { "message": "hello" } }, "error": null, "createdAt": "2026-05-21T10:00:00.000Z", "startedAt": "2026-05-21T10:00:00.100Z", "finishedAt": "2026-05-21T10:00:00.150Z" } ``` ### Authentication When `accessToken` is set, protected routes accept: - Header: `Authorization: Bearer ` - Query: `?token=` `GET /health` remains public for probes. ### Performance & Reliability - Increase **`maxConcurrentTasks`** to drain the queue faster — avoid overloading downstream targets if **`fetch`** is enabled. - **`timeoutMs`** on wait endpoints is clamped between safe bounds from input. - Actor run lifetime is still bounded by your Apify plan timeout. - Edge proxies may impose limits below your configured `timeoutMs`. ### Use Cases | Scenario | Fits when… | |----------|------------| | **Long interactions without blocking callers** | Accept job now, finalize later via poll/wait | | **Parallel fan-out demos** | Many **echo** or bounded **sleep** jobs to prove queues | | **Controlled enrichment fetch** | You explicitly enable **`allowFetchTasks`** and lock down **`accessToken`** | | **Webhook receivers** | Your external service POSTs tasks to the Container URL | For bulk REST integrations requiring bodies, multipart form, retries, or OAuth — extend this codebase or wrap a dedicated upstream service. ### Integrations & API - Call the Container URL from **curl**, **Postman**, **Zapier**, **Make**, or your backend - Read finished jobs from the Apify **dataset API** - Chain with schedules: keep a long-lived run or restart per batch depending on your pattern - Use Output schema links in Console for **Container API** and **dataset** URLs after each run ### FAQ #### Is this async or synchronous? Both. Jobs are async by default (`POST /tasks` → `202`). Wait endpoints block on the same Actor process until the job finishes or times out. #### Does the Actor retry failed fetch tasks? No automatic retry inside the worker. Clients may submit a new `POST /tasks` if needed. #### Can I send PUT/PATCH bodies or multipart forms? The shipped **`fetch`** path forwards **method + headers** only — extend `executeTask` if you need bodies or uploads. #### How do I secure outbound fetch? Keep **`allowFetchTasks: false`** unless required. Always set **`accessToken`** and restrict who can reach the Container URL. #### Where do results go? Each finished job is **`pushData`**’d to the default dataset. Pending/running jobs exist only in memory until they complete. #### Why is my Container URL empty locally? Set `ACTOR_WEB_SERVER_URL` (see local development below). On Apify, the URL appears on the run page when the container web server is enabled. ### SEO Keywords (high-intent) async api runner apify\ custom run async endpoint\ apify container web server\ background job queue actor\ async task endpoint\ apify express api actor\ long running api apify\ dataset job audit apify ### Why Choose This Actor? - Native Apify container URL — no separate hosting for the API layer - **202 Accepted** job model with optional blocking waits - Concurrency control built in - Dataset row per job for automation and auditing - Optional auth and fetch guardrails ### Limitations | Item | Detail | |------|--------| | **Run lifetime** | Bounded by Actor plan / max run duration | | **No built-in retry/backoff** | Add in clients or extend worker | | **Fetch semantics** | Single `fetch` call; body preview capped at 50k chars | | **In-memory queue** | Jobs are lost if the run aborts before completion | | **Not a full integration bus** | Echo/sleep/guarded fetch only in stock code | ### Running Locally **Apify CLI (recommended):** ```bash cd custom-run-async-endpoint npm install apify run ``` **Manual Node (Windows PowerShell example):** ```powershell cd custom-run-async-endpoint npm install Remove-Item Env:APIFY_IS_AT_HOME -ErrorAction SilentlyContinue $env:APIFY_LOCAL_STORAGE_DIR = "$PWD\storage" $env:CRAWLEE_STORAGE_DIR = "$PWD\storage" $env:CRAWLEE_PURGE_ON_START = "0" $env:ACTOR_WEB_SERVER_PORT = "4321" $env:ACTOR_WEB_SERVER_URL = "http://127.0.0.1:4321" npm start ``` Then open `http://127.0.0.1:4321/health`. ### Deploy to Apify 1. Push from `custom-run-async-endpoint/` (`apify push` or Git integration). 2. Enable **Container web server** in Actor settings. 3. Build and start a run → copy **Container URL** → verify **`GET /health`**. 4. Use **Output** schema links for Container API and dataset access. ### Get Started Enable the container web server, start a run, submit your first **`echo`** task to **`POST /tasks`**, and inspect job rows in the dataset — then scale up with concurrency, auth, and wait endpoints as needed. # Actor input Schema ## `accessToken` (type: `string`): If set, clients must send Authorization: Bearer or ?token= on API routes (except GET /health). ## `maxConcurrentTasks` (type: `integer`): Maximum number of tasks executing at once. ## `defaultWaitTimeoutMs` (type: `integer`): Default maximum time for POST /tasks/:jobId/wait and POST /run-and-wait when the client does not pass timeoutMs. ## `maxWaitTimeoutMs` (type: `integer`): Upper bound for any wait timeout from query/body (protects the process from absurd values). ## `allowFetchTasks` (type: `boolean`): If true, tasks of type "fetch" may call arbitrary URLs from the Actor (use only with accessToken and trusted callers). ## `maxSleepMs` (type: `integer`): Cap for type "sleep" tasks (testing / long-running simulation). ## Actor input object example ```json { "maxConcurrentTasks": 5, "defaultWaitTimeoutMs": 600000, "maxWaitTimeoutMs": 3600000, "allowFetchTasks": false, "maxSleepMs": 3600000 } ``` # Actor output Schema ## `containerApi` (type: `string`): Base URL for POST /tasks, GET /tasks/:jobId, and wait endpoints while the Actor run is active. ## `jobResults` (type: `string`): Dataset rows written when each job finishes (completed or failed). # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("sovanza.inc/custom-run-async-endpoint").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("sovanza.inc/custom-run-async-endpoint").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call sovanza.inc/custom-run-async-endpoint --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=sovanza.inc/custom-run-async-endpoint", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Custom Run Async Endpoint", "description": "Custom Run Async Endpoint runs an async HTTP API inside an Apify Actor container. Submit jobs, poll status, wait for results, control concurrency, protect routes with Bearer auth, and save completed job outputs to the dataset.", "version": "0.0", "x-build-id": "CfSe0gmYpMEqcloxo" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/sovanza.inc~custom-run-async-endpoint/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-sovanza.inc-custom-run-async-endpoint", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/sovanza.inc~custom-run-async-endpoint/runs": { "post": { "operationId": "runs-sync-sovanza.inc-custom-run-async-endpoint", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/sovanza.inc~custom-run-async-endpoint/run-sync": { "post": { "operationId": "run-sync-sovanza.inc-custom-run-async-endpoint", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "accessToken": { "title": "Access token (optional)", "type": "string", "description": "If set, clients must send Authorization: Bearer or ?token= on API routes (except GET /health)." }, "maxConcurrentTasks": { "title": "Max concurrent tasks", "minimum": 1, "maximum": 50, "type": "integer", "description": "Maximum number of tasks executing at once.", "default": 5 }, "defaultWaitTimeoutMs": { "title": "Default wait timeout (ms)", "minimum": 1000, "maximum": 86400000, "type": "integer", "description": "Default maximum time for POST /tasks/:jobId/wait and POST /run-and-wait when the client does not pass timeoutMs.", "default": 600000 }, "maxWaitTimeoutMs": { "title": "Hard cap for wait (ms)", "minimum": 5000, "maximum": 86400000, "type": "integer", "description": "Upper bound for any wait timeout from query/body (protects the process from absurd values).", "default": 3600000 }, "allowFetchTasks": { "title": "Allow fetch tasks", "type": "boolean", "description": "If true, tasks of type \"fetch\" may call arbitrary URLs from the Actor (use only with accessToken and trusted callers).", "default": false }, "maxSleepMs": { "title": "Max sleep per sleep task (ms)", "minimum": 0, "maximum": 86400000, "type": "integer", "description": "Cap for type \"sleep\" tasks (testing / long-running simulation).", "default": 3600000 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```