Video Auto Reframe Crop Keyframes

Generate video auto reframe instructions from authorized direct video downloads. The Actor detects faces locally, tracks their positions without recognizing identities, estimates speaker activity from mouth motion and audio energy, and exports normalized crop keyframes for vertical podcast editing.

This is a crop-instruction generator, not a video renderer. No face embeddings, identity lookup, transcription or external vision API is used. Review the keyframes before publishing an edit: speaker estimates are heuristic, especially with overlapping speech, occlusion, fast cuts or background music. Evidence warmup, switch hysteresis and camera smoothing can delay correct framing for several seconds.

Who is it for?

• Podcast editors preparing vertical interview clips.
• Creators automating portrait crops for talking-head videos.
• Developers feeding normalized rectangles into their own rendering pipeline.

Why use video auto reframe keyframes?

The output separates spatial crop instructions from video rendering. You can reuse your own renderer, retain the original media, and inspect whether a keyframe used speaker evidence or fallback framing. Coordinates are resolution-independent and preserve the requested pixel aspect ratio.

Getting started

1. Provide an anonymous direct MP4/MOV, WebM/Matroska or AVI URL. A video web page is not a media URL.
2. Confirm you own the video or have permission to analyze it by setting authorized to true.
3. Choose an aspect ratio, sample rate and maximum prefix duration.
4. Run the Actor and open the default dataset.
5. Export JSON and use the keyframes array in your editor or renderer.

The prefilled NASA interview URL is a public NASA media example, not a recommendation to analyze arbitrary third-party footage. NASA is not affiliated with this Actor. Example media availability can change.

Input example

{
  "videoUrls": ["https://example.com/authorized-interview.mp4"],
  "authorized": true,
  "maxAnalysisSeconds": 120,
  "sampleFps": 4,
  "targetAspectRatio": "9:16"
}

Replace the illustrative URL with your own reachable direct download.

Input parameters

Field	Behavior
videoUrls	1–5 direct URLs. Exact duplicates are processed once, sequentially. No login, page extraction, playlists or custom headers.
authorized	Required explicit true; default false. Confirm permission for local face-location/motion analysis.
maxAnalysisSeconds	Integer 1–600 per distinct video, default 120. Only the beginning is analyzed; longer media gets a truncation warning.
sampleFps	Integer 2–8, default 4. One keyframe per sampled frame.
targetAspectRatio	9:16 (default), 1:1, 4:5, or 16:9.

Every file is downloaded in full before analyzing its bounded prefix. Downloads must finish within 90 seconds and cannot exceed 250 MiB. Maximum source size is 8,294,400 total pixels, with each dimension between 32 and 3840 pixels. Non-square-pixel video is rejected; normalize it first. Local analysis has a 600-second deadline per video. These limits bound runtime; there is no unlimited mode.

Output fields

One dataset row represents one distinct successfully analyzed video.

Field	Meaning
sourceUrl	Input URL with query strings removed; signed links may therefore no longer resolve.
width, height	Orientation-corrected source dimensions.
durationSeconds	Full source duration.
analyzedSeconds	Actual prefix analyzed, capped by input.
audioAvailable	Whether an audio stream was decoded.
sampleFps, sampleCount	Analysis rate and decoded frame count.
faceSampleCount	Sample frames containing at least one detected face.
targetAspectRatio	Requested crop aspect.
downloadBytes	Full media download size.
keyframes	Timestamped normalized rectangles, ephemeral track IDs and framing evidence.
warnings	Truncation, missing audio or no-face disclosures.
analyzedAt	Completion timestamp.

Keyframe example

These spatial values come from a local interview test; its URL is replaced with an illustrative authorized-media URL.

{
  "sourceUrl": "https://example.com/authorized-interview.mp4",
  "width": 640,
  "height": 360,
  "durationSeconds": 1165.6,
  "analyzedSeconds": 20,
  "audioAvailable": true,
  "targetAspectRatio": "9:16",
  "sampleFps": 4,
  "sampleCount": 80,
  "faceSampleCount": 80,
  "keyframes": [
    {
      "timeSeconds": 0,
      "x": 0.351063,
      "y": 0,
      "width": 0.316406,
      "height": 1,
      "trackId": 1,
      "framingMode": "visual-fallback",
      "speakerScore": null
    }
  ],
  "warnings": ["Analysis truncated at requested limit."]
}

The example abbreviates the keyframe array and omits completion/download metadata. A 20-second video sampled at 4 fps normally yields 80 keyframes.

Coordinate conventions

x, y, width and height are fractions of the orientation-corrected source. Multiply horizontal values by source width and vertical values by source height. x and y mark the crop's upper-left corner. Rectangles remain inside the source. Timestamps start at zero; this Actor does not select highlight ranges or offset the analysis window.

Smooth motion is applied to crop centers. Interpolate between keyframes in your renderer. Integer-pixel rounding should clamp the final rectangle to the source edges. Portrait source media is cropped rather than padded. A crop cannot always retain all faces in a multi-person shot.

Speaker evidence and fallbacks

• audio-visual: audible energy plus sufficiently strong mouth-region motion; audio-envelope correlation weights the score. A sustained challenger can replace the previous track.
• visual-fallback: speaker evidence is ambiguous, silent or unavailable. Retain the prior visible face, otherwise the largest detected face.
• center-fallback: no face is detected. Keep a centered or previously held crop with trackId: null.

speakerScore is a heuristic evidence score, not a probability or speaker identity. It is null for fallback frames. Track IDs only connect geometric positions within the same video, can change after occlusion/cuts, and must not be used to identify people across videos.

No-face videos still produce usable centered crop instructions with a warning. They do not demonstrate successful active-speaker detection.

How much does it cost to generate video crop keyframes?

The current configured price is a $0.005 start fee per run plus one item event for each successfully analyzed distinct video. Keyframes have no separate charge. Failed videos do not incur an item event; an earlier successful video in a failed batch remains billable.

Apify Store spend tier	Price per analyzed video
FREE	$0.5146
BRONZE	$0.44748
SILVER	$0.34903
GOLD	$0.26849
PLATINUM	$0.26849
DIAMOND	$0.26849

At BRONZE, estimated Actor event charges are $0.45248 for one video and $2.2424 for five videos in one run. Spend tiers reflect aggregate qualifying monthly Apify Store spend, not this Actor's private volume bands. Check the Store's active price before running. Estimates are not guaranteed invoices; refunds, disputes, fraud, taxes, corrections and contractual clawbacks can affect billing or payouts.

Integrations

• Export JSON to your rendering worker and translate normalized rectangles into pixel coordinates.
• Join this output with independently generated transcript timestamps to frame a selected spoken passage. This Actor does not provide transcripts.
• Schedule analysis of your own new clip URLs. It does not monitor directories, discover videos or compare past runs automatically.
• Use CSV for summary columns, but prefer JSON for nested keyframes.

API usage

Use an Apify token in an environment variable rather than placing it in media URLs.

cURL

curl -X POST "https://api.apify.com/v2/acts/automation-lab~video-speaker-crop-keyframes/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"videoUrls":["https://example.com/authorized-interview.mp4"],"authorized":true,"maxAnalysisSeconds":120}'

Retain the returned run and dataset IDs. Poll that same run for terminal status, then fetch dataset items; IDs are not results.

JavaScript

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/video-speaker-crop-keyframes').call({
  videoUrls: ['https://example.com/authorized-interview.mp4'], authorized: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems({ limit: 5 });

Python

import os
from apify_client import ApifyClient
client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/video-speaker-crop-keyframes').call(run_input={
    'videoUrls': ['https://example.com/authorized-interview.mp4'], 'authorized': True,
})
rows = client.dataset(run['defaultDatasetId']).list_items(limit=5).items

MCP setup and bounded results

Claude Code

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/video-speaker-crop-keyframes"

Claude Desktop, Cursor and VS Code

Use the equivalent HTTP server configuration supported by your client:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/video-speaker-crop-keyframes"
    }
  }
}

Authenticate according to your client's Apify setup. Discover actual names and argument schemas with read-only tools/list. The Actor-scoped endpoint also supplies run/data helper tools; it is not exactly one tool.

Example prompt: “For this video I own, analyze its first 30 seconds and return a 9:16 crop summary, separating speaker estimates from fallback frames.”

For an authorized invocation, start once and retain its run ID. call-actor waits at most 45 seconds; use get-actor-run for the same run with 2/4/8-second backoff capped at 10 and a 120-second consumer deadline. Set the client request timeout longer than the selected wait plus transport margin. On timeout, recover the known run; do not start another run merely to poll. At deadline report pending and the run ID.

After success, fetch bounded get-dataset-items pages, starting at offset 0. A useful consumer limit is five video rows and 64 KiB total serialized UTF-8 content admitted to model context, whichever comes first. A keyframe row can be larger: enforce the byte cap host-side before model injection, summarize/omit with disclosure, and keep complete exports outside context. If your client cannot intercept oversized pages, do not claim a hard byte guarantee. Report source offsets and truncation; terminal failures may leave partial rows. No numeric token-savings guarantee is made.

Legality, privacy, AI and responsible use

Use only videos you own or are authorized to process. Face locations and motion are potentially personal data even without recognition. Do not use this Actor for identification, surveillance, sensitive profiling or decisions about people.

The MIT-licensed OpenCV YuNet model runs on CPU inside the Actor container. Media/audio frames stay local to that run; no vision provider receives them, and no model training occurs. Working downloads and audio files are deleted after each video, including failures under normal cleanup. Container-local data can remain until container disposal after a forced kill.

Dataset crop geometry and the sanitized source path remain in your Apify run storage until you delete that storage or your configured Apify retention removes it. Apify also stores input and logs under your account; private signed URL queries in input are not erased by output sanitization. Do not share run input/storage publicly. We do not create persistent face images, embeddings or cross-video identity profiles.

Failed operations send sanitized diagnostic input, exceptions and actor/build/run IDs to our private GlitchTip service for repair; secret fields and URL queries are removed and reports are retained for 30 days. No raw video or audio is sent there. This Actor uses standard Apify terms and is not affiliated with NASA, OpenCV or your media source.

Limitations and troubleshooting

Speaker-following is best on stable talking-head interviews. Motion, low resolution, tiny faces, occlusion, rapid cuts, similar motion among speakers, music and overlapping speech can produce fallback or incorrect crops. Review edits manually. Detection uses frames downscaled to at most 320 pixels on their longest edge; small faces may be missed. Up to 12 faces are considered per sampled frame; this is not crowd analytics.

An unsupported download, private destination, decode failure or exceeded deadline fails the run rather than returning false success. Completed earlier videos remain in a partial dataset. Signed URLs must remain valid for the download; a stored query-free source link is provenance, not a reusable signed credential.

FAQ

Does it render a vertical video? No. It returns crop rectangles for your renderer.

Does it identify the speaker by name? No. IDs are ephemeral geometric tracks; there is no recognition or embedding storage.

Can it analyze a YouTube page? No. Supply an authorized anonymous direct video file instead.

Why does my long video stop early? Analysis is limited to the first maxAnalysisSeconds; a warning reports truncation. Download limits apply to the whole file.

Why is there only fallback output? Faces/audio may be absent or evidence ambiguous. Inspect framingMode and warnings instead of treating every crop as a detected speaker.

Related workflows and support

This is a standalone local geometry utility. No other Actor, transcription service, proxy subscription or paid model key is required. If you need transcripts or rendering, choose those as separate pipeline stages rather than assuming they are included.

Report problems through the Actor's Apify Store issue channel with a safe reproduction input, run ID, expected crop behavior and permission-safe sample. Do not include credentials or private videos in a public report.