screen-analysis — screen signals for a composite source

Sibling of source-layout: where source-layout is pure geometry (panels, borders, layout states), screen-analysis adds what geometry alone cannot give — a face inside a panel (YuNet ONNX, MIT, two-scale: full-frame pass + a cropped/upscaled pass for small panels), text-line boxes (pure JS, no OCR), slide/region-change events (16x16 pHash + largest-changed-block diff), a cursor blob, and borderless-webcam overlay candidates. Signals only — no panel role or layout decision is made here (that is the core's job).

Reads an optional SourceLayoutDoc (from source-layout, inline via options.layout or fetched via options.layout_url) to know where the panels are; with no doc the whole frame is treated as one panel full.

Input

{
  "op": "screen",                      // or "capabilities"
  "source": "https:// … | kv:<storeId>/<key>",
  "options": {
    "start_sec": 0, "duration_sec": null,
    "sample_fps": 1, "analysis_width": 480,
    "layout": null,        // inline SourceLayoutDoc, or
    "layout_url": null,    // fetch it from here
    "onnx": true           // false forces the pure-geometry-only path
  },
  "output": { "signed_upload_url": null },
  "cleanup": "on_success",
  "callback": { "url": "…", "secret_header": { "name": "x-…", "value": "…" } }
}

Output

Artifact screen (screen.json): a ScreenSignals v1 document that parses with ScreenSignalsSchema (packages/edit-studio-core/src/editplan/screenSignals.ts):

{
  "version": 1,
  "probe": { "name": "screen-analysis", "version": "1.2.0" },
  "source": { "w": 1920, "h": 1080, "analysisWidth": 480 },
  "windows": [
    { "t0": 0, "t1": 2, "stateId": "s0", "panels": [
      { "panelId": "p0", "face": null, "varTemporal": 0.6, "flatness": 0.1,
        "edgeDensity": 0.2, "noiseHF": 1.8, "straightEdges": 0.05,
        "textBoxes": [], "cursor": null }
    ] }
  ],
  "overlayCandidates": [
    { "id": "ov0", "rect": { "x": 0.7, "y": 0.6, "w": 0.22, "h": 0.33 },
      "t0": 4, "t1": 20, "confidence": 0.8, "cues": ["noiseHF", "soft-border"] }
  ],
  "screenEvents": [
    { "kind": "slide-change", "atSec": 8, "panelId": "p1", "magnitude": 0.1, "rect": null }
  ]
}

All rects are 0..1 of the SOURCE frame. Deterministic: no randomness, no wall-clock in the algorithm — the same input produces the same JSON byte for byte.

Models

• face_detection_yunet_2023mar.onnx (YuNet, MIT) — downloaded at image build time into /models (MODELS_DIR). Missing file or onnx:false → face: null everywhere, same as "no face found"; every other signal is unaffected. Run op: "capabilities" to see model presence.
• No text-detection model ships (THIRD_PARTY.md explains why); text boxes come from a pure-JS detector.

Local run

npm install
MODELS_DIR=/tmp/models node src/main.js   # reads APIFY_INPUT_* env/local input per Apify CLI conventions
npm test                                  # synthetic fixtures generated with ffmpeg under /tmp, not committed

Cost

~1 fps geometry pass plus an optional ONNX face pass on panels; CPU only, no GPU. Budget target: within +50% of the sibling source-layout compute time per source-minute at the same analysis_width. Measured locally on a 60s 854x480 clip, sample_fps=1: ~2.8s/source-minute with onnx:false, ~2.9s/ source-minute with the YuNet model loaded (onnx:true) — well inside budget.

Deployment status

Not deployed yet. This actor exists only as source in this repo (apify-actor/apify-cloud-tools/screen-analysis); it has an entry in apify-actor/targets.json (key screenAnalysis) with an empty actorId placeholder — no Apify actor has been created/pushed for it. Do not run any push/deploy tooling against it until an actorId is assigned.

Probe history (probe.version)

Bump when a measured signal changes meaning — screen_signals_v2 is keyed by (project, probe_version) so new runs never overwrite old documents.

• 1.3.2 (08/10/2026): per-line rows inside each text box (

  textBoxes[].rows = [{y, h}]
  , frame-normalised like rect). The row projection already finds every line band before grouping them into a paragraph box; the studio needs the GAPS between lines so a zoom crop's top/bottom edge never slices a line in half (real 360p lecture: the whole slide was one 5-line box, the crop opened on half a line). Also: features.face now means the face runtime REALLY loaded (model file + onnxruntime-node); when it did not, faceDisabledReason says why instead of silently reporting "no face" on every panel.
• 1.3.1 (08/10/2026): layout state looked up on the source timeline. The layout document's states are absolute source seconds, and so is every window t0; 1.0–1.3.0 looked the state up at t0 - start_sec (shard-relative), so every shard after the first fell back to the FIRST state of the document — panel boxes, text boxes, region-change rects and per-panel faces were all measured on another layout's panels (real lecture 08/10: 492 of 585 windows stamped s0; the smart zoom cropped half of every text line away). Windows now carry the state that really covers them; shards starting at 0 are unchanged. Regression test: sharded fixture with two states.
• 1.3.0 (08/10/2026): focus windows. options.windows (

  [{start_sec, duration_sec}]
  , source timeline) lists the spans worth measuring; frames outside them are skipped entirely (no gray/Sobel/ONNX) and an open 2-second window is closed at the gap so no window bridges a skipped stretch. Shards are joined to keep the run count low and therefore cover the silence between clips — on a real 69-minute lecture 6 shards spanned 2389 s for 20 clip windows totalling 1044 s, so 56 % of the ONNX work measured seconds nobody used. The pipeline passes clip windows padded by 4 s each side (focusWindowsFor). No windows ⇒ whole shard, byte-identical to 1.2.0. timings.skippedFrames reports the saving. Same release, later build (no probe bump — output byte-identical): ONNX Runtime pinned to intraOpNumThreads: 1 and ffmpeg to -threads 2. A 4 GB actor gets about one vCPU, but ORT and ffmpeg size their thread pools from the HOST core count, so the container burst past its quota and was throttled by the cgroup (cpuMaxUsage 600 %, 165 ms per face pass on the cloud vs 21 ms locally). Measured on the same lecture: 420 s → 96 s wall, $0.204 → $0.049, identical 585 windows / 30 overlays / 170 events. Decode is still done for the whole shard; only the per-frame work is skipped outside focus windows.
• 1.2.0 (08/10/2026): nested panels are holes. When the layout document stacks a small camera panel inside a full-frame screen panel (source-layout ≥ 1.5 emits the borderless webcam that way), every measurement of the outer panel — face, motion share, slide hash, region change — skips the inner box: the screen is the screen, the person is the person. Slide-change now needs real evidence besides a hash jump (> 18 bits): changed blocks spread over ≥ 60 % of the panel, or ≥ 25 % of the area replaced, or ≥ 48 bits — a chart tick, a typed line or a demo frame is a region-change, not a new slide (1.1.0 called both "slide"). Event trackers are keyed per (layout state, panel), so a layout change no longer fabricates a slide-change at the cut. slide-change events carry areaShare.
• 1.1.0 (07/10/2026): varTemporal is the motion share — fraction of panel pixels whose gray changes by more than 12 between consecutive samples, averaged over the window (0..1). 1.0.0 measured the variance of the panel's mean brightness, which stays flat while a person sways or code scrolls, so every panel looked still to the role resolver. Windows are split at layout state changes (a 2 s window no longer straddles a cut, which used to paint the old state's screen panel "person" for a second around every change). Borderless-webcam boxes are tightened to the pixels that wobble at ≥ 40 % of the level inside the face box (codec ringing around the insert no longer inflates the box) and are no longer unioned with the detector's face box.
• 1.0.0 (07/10/2026): first version (geometry per panel × 2 s window, YuNet faces, text boxes, cursor, pHash slide events, borderless overlay candidates).