Custom Run Async Endpoint

Async API Runner – Custom Endpoint Execution & Background Processing

Run asynchronous workloads behind a hosted HTTP API on your Actor’s container web server URL. Submit jobs, poll for status, or wait longer than typical synchronous proxies allow — ideal when you need queued execution, controlled concurrency, and structured results in the Apify dataset without building extra infrastructure.

What is Custom Run Async Endpoint and How Does It Work?

Custom Run Async Endpoint is an Apify Actor that runs a long-lived Express HTTP server inside your Actor container. Clients call the Container URL to enqueue work; an internal worker pool executes tasks with a configurable concurrency cap; each finished job is written as one dataset row for audit and downstream automation.

This Actor is designed for:

• Developers building async handoffs on Apify
• Automation engineers who need 202 Accepted job submission
• Teams that want dataset-auditable task outcomes
• Integration prototypes with echo, sleep, or guarded fetch workloads

What this repo is: src/main.js implements job enqueue, an internal queue (maxConcurrentTasks), optional Bearer auth, blocking wait endpoints, and dataset export per job.

What it is not: a universal “retry any REST method with arbitrary JSON/form bodies” integration engine. Tasks are echo, sleep, or (optionally) fetch wrappers — extend the codebase or call from your own service for full REST orchestration.

Why Use This Async API Runner?

Use this Actor when:

• Upstream gateways time out before your work finishes
• You need async submission with optional blocking wait on the same run
• You want to cap parallel work with maxConcurrentTasks
• You need a dataset trail of every completed or failed job
• You already run on Apify and want a container URL without separate hosting

➡️ Best for controlled, auditable background-style processing inside an Apify run — not a detached serverless fleet.

Core Capabilities

Capability	Description
Async submission	POST /tasks returns jobId immediately (202); work continues if the client disconnects
Concurrency pool	maxConcurrentTasks (1–50) limits parallel workers
Blocking waits	POST /tasks/:jobId/wait and POST /run-and-wait poll until terminal status or timeout
Optional Bearer auth	accessToken in input → protected routes require Authorization: Bearer … or ?token= (except /health)
Guarded fetch	Outbound HTTP fetch tasks only when allowFetchTasks is true (off by default)
Dataset audit	Each terminal job pushes one row with status, task, result/error, timestamps

Retries: automatic HTTP retry/backoff is not implemented in main.js. Add retry logic in clients calling this API or extend the worker.

Execution Flow

1. Enable Container web server in Actor settings; Apify assigns ACTOR_WEB_SERVER_PORT and a public Container URL.
2. HTTP server listens on that port (default 4321 locally).
3. Client POST /tasks with task JSON → job pending → worker running → completed or failed.
4. Terminal jobs call Actor.pushData with jobId, task, result/error, timestamps.
5. Wait endpoints poll every ~200 ms until done or timeout (clamped by input).

HTTP Routes

Method	Path	Description
GET	/health	Liveness { ok, service, activeTasks, queued } — no auth
GET	/	Discovery JSON (routes + optional containerUrl)
POST	/tasks	Body = task JSON → 202 { jobId, getUrl, waitUrl }
GET	/tasks/:jobId	Full job snapshot
POST	/tasks/:jobId/wait	Block until terminal or timeout (timeoutMs query/body)
POST	/run-and-wait	Enqueue task + block until terminal or timeout

Task JSON (Supported Workloads)

Default type is echo if omitted.

type	Example body	Behaviour
echo	{ "type": "echo", "payload": { … } }	Returns structured echo result (safe demos)
sleep	{ "type": "sleep", "ms": 5000 }	Async delay capped by maxSleepMs
fetch	{ "type": "fetch", "url": "https://…", "method": "GET", "headers": { } }	Outbound fetch() only if allowFetchTasks is true. SSRF risk — use accessToken and trusted callers only. No configurable request body in shipped code.

How to Use on Apify

Using the Actor

1. Go to Custom Run Async Endpoint on the Apify platform.
2. Enable Container web server in Actor Settings → Container web server.
3. Configure input (optional accessToken, concurrency, wait timeouts, allowFetchTasks).
4. Start a run and copy the Container URL from the run page.
5. Call GET /health, then POST /tasks with your task JSON.
6. Poll GET /tasks/:jobId, block with POST /tasks/:jobId/wait, or use POST /run-and-wait.
7. Open the Dataset tab (or Output schema links) for finished job rows.

Input Configuration

{
  "accessToken": "your-secret-token",
  "maxConcurrentTasks": 5,
  "defaultWaitTimeoutMs": 600000,
  "maxWaitTimeoutMs": 3600000,
  "allowFetchTasks": false,
  "maxSleepMs": 3600000
}

Field	Description
accessToken	Optional shared secret (secret input). Protects all routes except GET /health.
maxConcurrentTasks	Max parallel workers (default 5, range 1–50).
defaultWaitTimeoutMs	Default wait window when client omits timeoutMs (default 600000).
maxWaitTimeoutMs	Hard cap for any wait timeout (default 3600000).
allowFetchTasks	Must be true to enable outbound fetch tasks (default false).
maxSleepMs	Maximum milliseconds for sleep tasks (default 3600000).

Output

The Actor exposes two output surfaces (see Output schema in Console):

Output	Description
Container API	Live base URL ({{run.containerUrl}}) while the run is active — use for /health, /tasks, wait routes
Job results	Default dataset — one row per completed or failed job

Dataset fields (per job)

Field	Description
jobId	UUID assigned at enqueue
status	completed or failed
task	Original task JSON
result	Structured result object when successful
error	Error message when failed
createdAt	When the job was enqueued
startedAt	When execution started
finishedAt	When the job finished

Example dataset row (echo task):

{
  "jobId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "completed",
  "task": { "type": "echo", "payload": { "message": "hello" } },
  "result": { "taskType": "echo", "payload": { "message": "hello" } },
  "error": null,
  "createdAt": "2026-05-21T10:00:00.000Z",
  "startedAt": "2026-05-21T10:00:00.100Z",
  "finishedAt": "2026-05-21T10:00:00.150Z"
}

Authentication

When accessToken is set, protected routes accept:

• Header: Authorization: Bearer <token>
• Query: ?token=<token>

GET /health remains public for probes.

Performance & Reliability

• Increase maxConcurrentTasks to drain the queue faster — avoid overloading downstream targets if fetch is enabled.
• timeoutMs on wait endpoints is clamped between safe bounds from input.
• Actor run lifetime is still bounded by your Apify plan timeout.
• Edge proxies may impose limits below your configured timeoutMs.

Use Cases

Scenario	Fits when…
Long interactions without blocking callers	Accept job now, finalize later via poll/wait
Parallel fan-out demos	Many echo or bounded sleep jobs to prove queues
Controlled enrichment fetch	You explicitly enable allowFetchTasks and lock down accessToken
Webhook receivers	Your external service POSTs tasks to the Container URL

For bulk REST integrations requiring bodies, multipart form, retries, or OAuth — extend this codebase or wrap a dedicated upstream service.

Integrations & API

• Call the Container URL from curl, Postman, Zapier, Make, or your backend
• Read finished jobs from the Apify dataset API
• Chain with schedules: keep a long-lived run or restart per batch depending on your pattern
• Use Output schema links in Console for Container API and dataset URLs after each run

FAQ

Is this async or synchronous?

Both. Jobs are async by default (POST /tasks → 202). Wait endpoints block on the same Actor process until the job finishes or times out.

Does the Actor retry failed fetch tasks?

No automatic retry inside the worker. Clients may submit a new POST /tasks if needed.

Can I send PUT/PATCH bodies or multipart forms?

The shipped fetch path forwards method + headers only — extend executeTask if you need bodies or uploads.

How do I secure outbound fetch?

Keep allowFetchTasks: false unless required. Always set accessToken and restrict who can reach the Container URL.

Where do results go?

Each finished job is pushData’d to the default dataset. Pending/running jobs exist only in memory until they complete.

Why is my Container URL empty locally?

Set ACTOR_WEB_SERVER_URL (see local development below). On Apify, the URL appears on the run page when the container web server is enabled.

SEO Keywords (high-intent)

async api runner apify
custom run async endpoint
apify container web server
background job queue actor
async task endpoint
apify express api actor
long running api apify
dataset job audit apify

Why Choose This Actor?

• Native Apify container URL — no separate hosting for the API layer
• 202 Accepted job model with optional blocking waits
• Concurrency control built in
• Dataset row per job for automation and auditing
• Optional auth and fetch guardrails

Limitations

Item	Detail
Run lifetime	Bounded by Actor plan / max run duration
No built-in retry/backoff	Add in clients or extend worker
Fetch semantics	Single fetch call; body preview capped at 50k chars
In-memory queue	Jobs are lost if the run aborts before completion
Not a full integration bus	Echo/sleep/guarded fetch only in stock code

Running Locally

Apify CLI (recommended):

cd custom-run-async-endpoint
npm install
apify run

Manual Node (Windows PowerShell example):

cd custom-run-async-endpoint
npm install
Remove-Item Env:APIFY_IS_AT_HOME -ErrorAction SilentlyContinue
$env:APIFY_LOCAL_STORAGE_DIR = "$PWD\storage"
$env:CRAWLEE_STORAGE_DIR = "$PWD\storage"
$env:CRAWLEE_PURGE_ON_START = "0"
$env:ACTOR_WEB_SERVER_PORT = "4321"
$env:ACTOR_WEB_SERVER_URL = "http://127.0.0.1:4321"
npm start

Then open http://127.0.0.1:4321/health.

Deploy to Apify

1. Push from custom-run-async-endpoint/ (apify push or Git integration).
2. Enable Container web server in Actor settings.
3. Build and start a run → copy Container URL → verify GET /health.
4. Use Output schema links for Container API and dataset access.

Get Started

Enable the container web server, start a run, submit your first echo task to POST /tasks, and inspect job rows in the dataset — then scale up with concurrency, auth, and wait endpoints as needed.