# Screen Analysis Probe for Composite Video (`andrew_babo/screen-analysis`) Actor

Signals a pure-geometry layout probe cannot give: faces inside panels (YuNet ONNX), text-line boxes (no OCR), slide/region-change events (pHash), cursor blob and borderless-webcam overlay candidates. Reads an optional SourceLayoutDoc from the source-layout actor. Signals only, no layout decision.

- **URL**: https://apify.com/andrew_babo/screen-analysis.md
- **Developed by:** [Andrew Babo](https://apify.com/andrew_babo) (community)
- **Stats:** 80 total users, 80 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## screen-analysis — screen signals for a composite source

Sibling of `source-layout`: where `source-layout` is pure geometry (panels,
borders, layout states), `screen-analysis` adds what geometry alone cannot
give — a face inside a panel (YuNet ONNX, MIT, two-scale: full-frame pass +
a cropped/upscaled pass for small panels), text-line boxes (pure JS, no OCR),
slide/region-change events (16x16 pHash + largest-changed-block diff), a
cursor blob, and borderless-webcam overlay candidates. **Signals only** — no
panel role or layout decision is made here (that is the core's job).

Reads an optional `SourceLayoutDoc` (from `source-layout`, inline via
`options.layout` or fetched via `options.layout_url`) to know where the
panels are; with no doc the whole frame is treated as one panel `full`.

### Input

```jsonc
{
  "op": "screen",                      // or "capabilities"
  "source": "https:// … | kv:<storeId>/<key>",
  "options": {
    "start_sec": 0, "duration_sec": null,
    "sample_fps": 1, "analysis_width": 480,
    "layout": null,        // inline SourceLayoutDoc, or
    "layout_url": null,    // fetch it from here
    "onnx": true           // false forces the pure-geometry-only path
  },
  "output": { "signed_upload_url": null },
  "cleanup": "on_success",
  "callback": { "url": "…", "secret_header": { "name": "x-…", "value": "…" } }
}
```

### Output

Artifact `screen` (`screen.json`): a `ScreenSignals` v1 document that parses
with `ScreenSignalsSchema` (`packages/edit-studio-core/src/editplan/screenSignals.ts`):

```jsonc
{
  "version": 1,
  "probe": { "name": "screen-analysis", "version": "1.2.0" },
  "source": { "w": 1920, "h": 1080, "analysisWidth": 480 },
  "windows": [
    { "t0": 0, "t1": 2, "stateId": "s0", "panels": [
      { "panelId": "p0", "face": null, "varTemporal": 0.6, "flatness": 0.1,
        "edgeDensity": 0.2, "noiseHF": 1.8, "straightEdges": 0.05,
        "textBoxes": [], "cursor": null }
    ] }
  ],
  "overlayCandidates": [
    { "id": "ov0", "rect": { "x": 0.7, "y": 0.6, "w": 0.22, "h": 0.33 },
      "t0": 4, "t1": 20, "confidence": 0.8, "cues": ["noiseHF", "soft-border"] }
  ],
  "screenEvents": [
    { "kind": "slide-change", "atSec": 8, "panelId": "p1", "magnitude": 0.1, "rect": null }
  ]
}
```

All rects are 0..1 of the SOURCE frame. Deterministic: no randomness, no
wall-clock in the algorithm — the same input produces the same JSON byte for
byte.

### Models

- `face_detection_yunet_2023mar.onnx` (YuNet, MIT) — downloaded at image
  build time into `/models` (`MODELS_DIR`). Missing file or `onnx:false` →
  `face: null` everywhere, same as "no face found"; every other signal is
  unaffected. Run `op: "capabilities"` to see model presence.
- No text-detection model ships (`THIRD_PARTY.md` explains why); text boxes
  come from a pure-JS detector.

### Local run

```bash
npm install
MODELS_DIR=/tmp/models node src/main.js   # reads APIFY_INPUT_* env/local input per Apify CLI conventions
npm test                                  # synthetic fixtures generated with ffmpeg under /tmp, not committed
```

### Cost

\~1 fps geometry pass plus an optional ONNX face pass on panels; CPU only, no
GPU. Budget target: within +50% of the sibling `source-layout` compute time
per source-minute at the same `analysis_width`. Measured locally on a 60s
854x480 clip, sample_fps=1: ~2.8s/source-minute with `onnx:false`, ~2.9s/
source-minute with the YuNet model loaded (`onnx:true`) — well inside budget.

### Deployment status

**Not deployed yet.** This actor exists only as source in this repo
(`apify-actor/apify-cloud-tools/screen-analysis`); it has an entry in
`apify-actor/targets.json` (key `screenAnalysis`) with an empty `actorId`
placeholder — no Apify actor has been created/pushed for it. Do not run any
push/deploy tooling against it until an `actorId` is assigned.

### Probe history (`probe.version`)

Bump when a measured signal changes meaning — `screen_signals_v2` is keyed by
`(project, probe_version)` so new runs never overwrite old documents.

- `1.3.2` (08/10/2026): **per-line rows inside each text box** (`textBoxes[].rows = [{y, h}]`, frame-normalised like `rect`). The row projection already finds every
  line band before grouping them into a paragraph box; the studio needs the GAPS
  between lines so a zoom crop's top/bottom edge never slices a line in half (real
  360p lecture: the whole slide was one 5-line box, the crop opened on half a line).
  Also: `features.face` now means the face runtime REALLY loaded (model file +
  `onnxruntime-node`); when it did not, `faceDisabledReason` says why instead of
  silently reporting "no face" on every panel.
- `1.3.1` (08/10/2026): **layout state looked up on the source timeline.** The
  layout document's states are absolute source seconds, and so is every window
  `t0`; 1.0–1.3.0 looked the state up at `t0 - start_sec` (shard-relative), so
  every shard after the first fell back to the FIRST state of the document —
  panel boxes, text boxes, region-change rects and per-panel faces were all
  measured on another layout's panels (real lecture 08/10: 492 of 585 windows
  stamped `s0`; the smart zoom cropped half of every text line away). Windows
  now carry the state that really covers them; shards starting at 0 are
  unchanged. Regression test: sharded fixture with two states.
- `1.3.0` (08/10/2026): **focus windows.** `options.windows` (`[{start_sec,
  duration_sec}]`, source timeline) lists the spans worth measuring; frames
  outside them are skipped entirely (no gray/Sobel/ONNX) and an open 2-second
  window is closed at the gap so no window bridges a skipped stretch. Shards are
  joined to keep the run count low and therefore cover the silence between
  clips — on a real 69-minute lecture 6 shards spanned 2389 s for 20 clip
  windows totalling 1044 s, so 56 % of the ONNX work measured seconds nobody
  used. The pipeline passes clip windows padded by 4 s each side
  (`focusWindowsFor`). No `windows` ⇒ whole shard, byte-identical to 1.2.0.
  `timings.skippedFrames` reports the saving.
  Same release, later build (no probe bump — output byte-identical): ONNX Runtime
  pinned to `intraOpNumThreads: 1` and ffmpeg to `-threads 2`. A 4 GB actor gets
  about one vCPU, but ORT and ffmpeg size their thread pools from the HOST core
  count, so the container burst past its quota and was throttled by the cgroup
  (`cpuMaxUsage` 600 %, 165 ms per face pass on the cloud vs 21 ms locally).
  Measured on the same lecture: 420 s → 96 s wall, $0.204 → $0.049, identical
  585 windows / 30 overlays / 170 events. Decode is still done for the whole
  shard; only the per-frame work is skipped outside focus windows.
- `1.2.0` (08/10/2026): **nested panels are holes.** When the layout document
  stacks a small camera panel inside a full-frame screen panel (source-layout
  ≥ 1.5 emits the borderless webcam that way), every measurement of the outer
  panel — face, motion share, slide hash, region change — skips the inner box:
  the screen is the screen, the person is the person. Slide-change now needs
  real evidence besides a hash jump (> 18 bits): changed blocks spread over
  ≥ 60 % of the panel, or ≥ 25 % of the area replaced, or ≥ 48 bits — a chart
  tick, a typed line or a demo frame is a `region-change`, not a new slide
  (1.1.0 called both "slide"). Event trackers are keyed per
  `(layout state, panel)`, so a layout change no longer fabricates a
  slide-change at the cut. `slide-change` events carry `areaShare`.
- `1.1.0` (07/10/2026): `varTemporal` is the **motion share** — fraction of
  panel pixels whose gray changes by more than 12 between consecutive samples,
  averaged over the window (0..1). 1.0.0 measured the variance of the panel's
  *mean* brightness, which stays flat while a person sways or code scrolls, so
  every panel looked still to the role resolver. Windows are **split at layout
  state changes** (a 2 s window no longer straddles a cut, which used to paint
  the old state's screen panel "person" for a second around every change).
  Borderless-webcam boxes are tightened to the pixels that wobble at ≥ 40 % of
  the level inside the face box (codec ringing around the insert no longer
  inflates the box) and are no longer unioned with the detector's face box.
- `1.0.0` (07/10/2026): first version (geometry per panel × 2 s window, YuNet
  faces, text boxes, cursor, pHash slide events, borderless overlay candidates).

# Actor input Schema

## `op` (type: `string`):

screen = sample the video and return screen signals (faces, text, events, overlay candidates). capabilities = return supported ops, features and model presence without running any work (no source needed).

## `source` (type: `string`):

https:// URL, or kv:<storeId>/<key> from an earlier run (e.g. the 480p proxy from video-audio-toolkit — this probe also analyses at 480 px wide by default).

## `options` (type: `object`):

{ start_sec, duration_sec } bound a shard (timestamps in the result are shifted back onto the source timeline); sample_fps (0.5-4, default 1); analysis_width (320-640, default 480); layout (inline SourceLayoutDoc) or layout_url (fetch it); onnx (default true — set false to force the pure-geometry-only path, e.g. when no model file is present).

## `output` (type: `object`):

{ signed_upload_url } to PUT the screen signals JSON straight into your own storage.

## `cleanup` (type: `string`):

on_success = keep only the screen artifact in the run's key-value store. always = also drop it after the signed upload. off = keep everything for debugging.

## `callback` (type: `object`):

{ url, secret_header: { name, value } } — the run result JSON is POSTed there when the run ends.

## Actor input object example

```json
{
  "op": "capabilities",
  "options": {
    "sample_fps": 1,
    "analysis_width": 480,
    "onnx": true
  },
  "cleanup": "on_success"
}
```

# Actor output Schema

## `results` (type: `string`):

Run result JSON: status, op, artifacts \[{name, kv_key, url, bytes}], meta (counts, feature flags, model presence), timings, errors.

## `resultRecord` (type: `string`):

The same result JSON stored as the RESULT record of the default key-value store.

## `screen` (type: `string`):

ScreenSignals v1: probe, source, windows\[], overlayCandidates\[], screenEvents\[] — parses with ScreenSignalsSchema (packages/edit-studio-core).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "op": "capabilities",
    "options": {
        "sample_fps": 1,
        "analysis_width": 480,
        "onnx": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("andrew_babo/screen-analysis").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "op": "capabilities",
    "options": {
        "sample_fps": 1,
        "analysis_width": 480,
        "onnx": True,
    },
}

# Run the Actor and wait for it to finish
run = client.actor("andrew_babo/screen-analysis").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "op": "capabilities",
  "options": {
    "sample_fps": 1,
    "analysis_width": 480,
    "onnx": true
  }
}' |
apify call andrew_babo/screen-analysis --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,andrew_babo/screen-analysis"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/T0M30M5ntqyXWnIAf/builds/16hWbzFy55DTswcI1/openapi.json