# Video Auto Reframe Crop Keyframes (`automation-lab/video-speaker-crop-keyframes`) Actor

Analyze authorized direct video URLs and export normalized speaker-following crop keyframes for vertical podcast editing, without identity recognition or persistent face embeddings.

- **URL**: https://apify.com/automation-lab/video-speaker-crop-keyframes.md
- **Developed by:** [Automation Lab](https://apify.com/automation-lab) (community)
- **Categories:** Videos, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $268.49 / 1,000 video analyzeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Video Auto Reframe Crop Keyframes

Generate **video auto reframe** instructions from authorized direct video downloads. The Actor detects faces locally, tracks their positions without recognizing identities, estimates speaker activity from mouth motion and audio energy, and exports normalized crop keyframes for vertical podcast editing.

This is a crop-instruction generator, not a video renderer. No face embeddings, identity lookup, transcription or external vision API is used. Review the keyframes before publishing an edit: speaker estimates are heuristic, especially with overlapping speech, occlusion, fast cuts or background music. Evidence warmup, switch hysteresis and camera smoothing can delay correct framing for several seconds.

### Who is it for?

- Podcast editors preparing vertical interview clips.
- Creators automating portrait crops for talking-head videos.
- Developers feeding normalized rectangles into their own rendering pipeline.

### Why use video auto reframe keyframes?

The output separates spatial crop instructions from video rendering. You can reuse your own renderer, retain the original media, and inspect whether a keyframe used speaker evidence or fallback framing. Coordinates are resolution-independent and preserve the requested pixel aspect ratio.

### Getting started

1. Provide an anonymous direct MP4/MOV, WebM/Matroska or AVI URL. A video web page is not a media URL.
2. Confirm you own the video or have permission to analyze it by setting `authorized` to `true`.
3. Choose an aspect ratio, sample rate and maximum prefix duration.
4. Run the Actor and open the default dataset.
5. Export JSON and use the `keyframes` array in your editor or renderer.

The prefilled NASA interview URL is a public NASA media example, not a recommendation to analyze arbitrary third-party footage. NASA is not affiliated with this Actor. Example media availability can change.

### Input example

```json
{
  "videoUrls": ["https://example.com/authorized-interview.mp4"],
  "authorized": true,
  "maxAnalysisSeconds": 120,
  "sampleFps": 4,
  "targetAspectRatio": "9:16"
}
```

Replace the illustrative URL with your own reachable direct download.

### Input parameters

| Field | Behavior |
| --- | --- |
| `videoUrls` | 1–5 direct URLs. Exact duplicates are processed once, sequentially. No login, page extraction, playlists or custom headers. |
| `authorized` | Required explicit `true`; default `false`. Confirm permission for local face-location/motion analysis. |
| `maxAnalysisSeconds` | Integer 1–600 per distinct video, default 120. Only the beginning is analyzed; longer media gets a truncation warning. |
| `sampleFps` | Integer 2–8, default 4. One keyframe per sampled frame. |
| `targetAspectRatio` | `9:16` (default), `1:1`, `4:5`, or `16:9`. |

Every file is downloaded in full before analyzing its bounded prefix. Downloads must finish within 90 seconds and cannot exceed 250 MiB. Maximum source size is 8,294,400 total pixels, with each dimension between 32 and 3840 pixels. Non-square-pixel video is rejected; normalize it first. Local analysis has a 600-second deadline per video. These limits bound runtime; there is no unlimited mode.

### Output fields

One dataset row represents one distinct successfully analyzed video.

| Field | Meaning |
| --- | --- |
| `sourceUrl` | Input URL with query strings removed; signed links may therefore no longer resolve. |
| `width`, `height` | Orientation-corrected source dimensions. |
| `durationSeconds` | Full source duration. |
| `analyzedSeconds` | Actual prefix analyzed, capped by input. |
| `audioAvailable` | Whether an audio stream was decoded. |
| `sampleFps`, `sampleCount` | Analysis rate and decoded frame count. |
| `faceSampleCount` | Sample frames containing at least one detected face. |
| `targetAspectRatio` | Requested crop aspect. |
| `downloadBytes` | Full media download size. |
| `keyframes` | Timestamped normalized rectangles, ephemeral track IDs and framing evidence. |
| `warnings` | Truncation, missing audio or no-face disclosures. |
| `analyzedAt` | Completion timestamp. |

### Keyframe example

These spatial values come from a local interview test; its URL is replaced with an illustrative authorized-media URL.

```json
{
  "sourceUrl": "https://example.com/authorized-interview.mp4",
  "width": 640,
  "height": 360,
  "durationSeconds": 1165.6,
  "analyzedSeconds": 20,
  "audioAvailable": true,
  "targetAspectRatio": "9:16",
  "sampleFps": 4,
  "sampleCount": 80,
  "faceSampleCount": 80,
  "keyframes": [
    {
      "timeSeconds": 0,
      "x": 0.351063,
      "y": 0,
      "width": 0.316406,
      "height": 1,
      "trackId": 1,
      "framingMode": "visual-fallback",
      "speakerScore": null
    }
  ],
  "warnings": ["Analysis truncated at requested limit."]
}
```

The example abbreviates the keyframe array and omits completion/download metadata. A 20-second video sampled at 4 fps normally yields 80 keyframes.

### Coordinate conventions

`x`, `y`, `width` and `height` are fractions of the orientation-corrected source. Multiply horizontal values by source width and vertical values by source height. `x` and `y` mark the crop's upper-left corner. Rectangles remain inside the source. Timestamps start at zero; this Actor does not select highlight ranges or offset the analysis window.

Smooth motion is applied to crop centers. Interpolate between keyframes in your renderer. Integer-pixel rounding should clamp the final rectangle to the source edges. Portrait source media is cropped rather than padded. A crop cannot always retain all faces in a multi-person shot.

### Speaker evidence and fallbacks

- `audio-visual`: audible energy plus sufficiently strong mouth-region motion; audio-envelope correlation weights the score. A sustained challenger can replace the previous track.
- `visual-fallback`: speaker evidence is ambiguous, silent or unavailable. Retain the prior visible face, otherwise the largest detected face.
- `center-fallback`: no face is detected. Keep a centered or previously held crop with `trackId: null`.

`speakerScore` is a heuristic evidence score, **not** a probability or speaker identity. It is null for fallback frames. Track IDs only connect geometric positions within the same video, can change after occlusion/cuts, and must not be used to identify people across videos.

No-face videos still produce usable centered crop instructions with a warning. They do not demonstrate successful active-speaker detection.

### How much does it cost to generate video crop keyframes?

The current configured price is a $0.005 start fee per run plus one `item` event for each successfully analyzed distinct video. Keyframes have no separate charge. Failed videos do not incur an item event; an earlier successful video in a failed batch remains billable.

| Apify Store spend tier | Price per analyzed video |
| --- | ---: |
| FREE | $0.5146 |
| BRONZE | $0.44748 |
| SILVER | $0.34903 |
| GOLD | $0.26849 |
| PLATINUM | $0.26849 |
| DIAMOND | $0.26849 |

At BRONZE, estimated Actor event charges are $0.45248 for one video and $2.2424 for five videos in one run. Spend tiers reflect aggregate qualifying monthly Apify Store spend, not this Actor's private volume bands. Check the Store's active price before running. Estimates are not guaranteed invoices; refunds, disputes, fraud, taxes, corrections and contractual clawbacks can affect billing or payouts.

### Integrations

- Export JSON to your rendering worker and translate normalized rectangles into pixel coordinates.
- Join this output with independently generated transcript timestamps to frame a selected spoken passage. This Actor does not provide transcripts.
- Schedule analysis of your own new clip URLs. It does not monitor directories, discover videos or compare past runs automatically.
- Use CSV for summary columns, but prefer JSON for nested keyframes.

### API usage

Use an Apify token in an environment variable rather than placing it in media URLs.

#### cURL

```bash
curl -X POST "https://api.apify.com/v2/acts/automation-lab~video-speaker-crop-keyframes/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"videoUrls":["https://example.com/authorized-interview.mp4"],"authorized":true,"maxAnalysisSeconds":120}'
```

Retain the returned run and dataset IDs. Poll that same run for terminal status, then fetch dataset items; IDs are not results.

#### JavaScript

```javascript
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/video-speaker-crop-keyframes').call({
  videoUrls: ['https://example.com/authorized-interview.mp4'], authorized: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems({ limit: 5 });
```

#### Python

```python
import os
from apify_client import ApifyClient
client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/video-speaker-crop-keyframes').call(run_input={
    'videoUrls': ['https://example.com/authorized-interview.mp4'], 'authorized': True,
})
rows = client.dataset(run['defaultDatasetId']).list_items(limit=5).items
```

### MCP setup and bounded results

#### Claude Code

```bash
claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/video-speaker-crop-keyframes"
```

#### Claude Desktop, Cursor and VS Code

Use the equivalent HTTP server configuration supported by your client:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/video-speaker-crop-keyframes"
    }
  }
}
```

Authenticate according to your client's Apify setup. Discover actual names and argument schemas with read-only `tools/list`. The Actor-scoped endpoint also supplies run/data helper tools; it is not exactly one tool.

Example prompt: “For this video I own, analyze its first 30 seconds and return a 9:16 crop summary, separating speaker estimates from fallback frames.”

For an authorized invocation, start once and retain its run ID. `call-actor` waits at most 45 seconds; use `get-actor-run` for the same run with 2/4/8-second backoff capped at 10 and a 120-second consumer deadline. Set the client request timeout longer than the selected wait plus transport margin. On timeout, recover the known run; do not start another run merely to poll. At deadline report pending and the run ID.

After success, fetch bounded `get-dataset-items` pages, starting at offset 0. A useful consumer limit is five video rows and 64 KiB total serialized UTF-8 content admitted to model context, whichever comes first. A keyframe row can be larger: enforce the byte cap host-side before model injection, summarize/omit with disclosure, and keep complete exports outside context. If your client cannot intercept oversized pages, do not claim a hard byte guarantee. Report source offsets and truncation; terminal failures may leave partial rows. No numeric token-savings guarantee is made.

### Legality, privacy, AI and responsible use

Use only videos you own or are authorized to process. Face locations and motion are potentially personal data even without recognition. Do not use this Actor for identification, surveillance, sensitive profiling or decisions about people.

The MIT-licensed OpenCV YuNet model runs on CPU inside the Actor container. Media/audio frames stay local to that run; no vision provider receives them, and no model training occurs. Working downloads and audio files are deleted after each video, including failures under normal cleanup. Container-local data can remain until container disposal after a forced kill.

Dataset crop geometry and the sanitized source path remain in your Apify run storage until you delete that storage or your configured Apify retention removes it. Apify also stores input and logs under your account; private signed URL queries in input are not erased by output sanitization. Do not share run input/storage publicly. We do not create persistent face images, embeddings or cross-video identity profiles.

Failed operations send sanitized diagnostic input, exceptions and actor/build/run IDs to our private GlitchTip service for repair; secret fields and URL queries are removed and reports are retained for 30 days. No raw video or audio is sent there. This Actor uses standard Apify terms and is not affiliated with NASA, OpenCV or your media source.

### Limitations and troubleshooting

Speaker-following is best on stable talking-head interviews. Motion, low resolution, tiny faces, occlusion, rapid cuts, similar motion among speakers, music and overlapping speech can produce fallback or incorrect crops. Review edits manually. Detection uses frames downscaled to at most 320 pixels on their longest edge; small faces may be missed. Up to 12 faces are considered per sampled frame; this is not crowd analytics.

An unsupported download, private destination, decode failure or exceeded deadline fails the run rather than returning false success. Completed earlier videos remain in a partial dataset. Signed URLs must remain valid for the download; a stored query-free source link is provenance, not a reusable signed credential.

### FAQ

**Does it render a vertical video?** No. It returns crop rectangles for your renderer.

**Does it identify the speaker by name?** No. IDs are ephemeral geometric tracks; there is no recognition or embedding storage.

**Can it analyze a YouTube page?** No. Supply an authorized anonymous direct video file instead.

**Why does my long video stop early?** Analysis is limited to the first `maxAnalysisSeconds`; a warning reports truncation. Download limits apply to the whole file.

**Why is there only fallback output?** Faces/audio may be absent or evidence ambiguous. Inspect `framingMode` and warnings instead of treating every crop as a detected speaker.

### Related workflows and support

This is a standalone local geometry utility. No other Actor, transcription service, proxy subscription or paid model key is required. If you need transcripts or rendering, choose those as separate pipeline stages rather than assuming they are included.

Report problems through the Actor's Apify Store issue channel with a safe reproduction input, run ID, expected crop behavior and permission-safe sample. Do not include credentials or private videos in a public report.

# Changelog

This Actor's version history is a separate document: https://apify.com/automation-lab/video-speaker-crop-keyframes/changelog.md

# Actor input Schema

## `videoUrls` (type: `array`):

1–5 anonymous HTTP(S) MP4/MOV, WebM/Matroska or AVI downloads on ports 80/443. Each download is capped at 250 MiB and 90 seconds; redirects are checked against private network access. No page URLs, playlists, credentials or login. Exact duplicate URLs are analyzed once. Source query strings are omitted from output.

## `authorized` (type: `boolean`):

Required explicit consent: true only when you own the video or have permission to perform automated face-location and motion analysis. Does not authorize identity recognition; no face embeddings are generated.

## `maxAnalysisSeconds` (type: `integer`):

Analyze the beginning of each distinct video, up to this many seconds (1–600, default 120). Longer videos produce an explicit truncation warning. Short videos stop at their actual end. The entire bounded media file must still be downloaded. Zero/unlimited is unsupported.

## `sampleFps` (type: `integer`):

Integer from 2 to 8 sampled frames per second, default 4. One keyframe per sampled frame; timestamps are relative to the original video start. Higher sampling increases compute and temporal detail, not a guarantee of correct speaker detection.

## `targetAspectRatio` (type: `string`):

Output rectangle pixel aspect ratio: 9:16 (default), 1:1, 4:5 or 16:9. Crop stays inside the orientation-corrected source; no padding or rendering is performed. A narrow crop may not contain every face.

## Actor input object example

```json
{
  "videoUrls": [
    "https://images-assets.nasa.gov/video/iss061m2627771232_Live_Interviews_Jeanette_Epps_191004/iss061m2627771232_Live_Interviews_Jeanette_Epps_191004~small.mp4"
  ],
  "authorized": true,
  "maxAnalysisSeconds": 20,
  "sampleFps": 4,
  "targetAspectRatio": "9:16"
}
```

# Actor output Schema

## `dataset` (type: `string`):

One row per distinct successfully analyzed video, including time-indexed crop rectangles.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "videoUrls": [
        "https://images-assets.nasa.gov/video/iss061m2627771232_Live_Interviews_Jeanette_Epps_191004/iss061m2627771232_Live_Interviews_Jeanette_Epps_191004~small.mp4"
    ],
    "authorized": true,
    "maxAnalysisSeconds": 20,
    "sampleFps": 4,
    "targetAspectRatio": "9:16"
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/video-speaker-crop-keyframes").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "videoUrls": ["https://images-assets.nasa.gov/video/iss061m2627771232_Live_Interviews_Jeanette_Epps_191004/iss061m2627771232_Live_Interviews_Jeanette_Epps_191004~small.mp4"],
    "authorized": True,
    "maxAnalysisSeconds": 20,
    "sampleFps": 4,
    "targetAspectRatio": "9:16",
}

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/video-speaker-crop-keyframes").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "videoUrls": [
    "https://images-assets.nasa.gov/video/iss061m2627771232_Live_Interviews_Jeanette_Epps_191004/iss061m2627771232_Live_Interviews_Jeanette_Epps_191004~small.mp4"
  ],
  "authorized": true,
  "maxAnalysisSeconds": 20,
  "sampleFps": 4,
  "targetAspectRatio": "9:16"
}' |
apify call automation-lab/video-speaker-crop-keyframes --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/video-speaker-crop-keyframes"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Mc3DZ8Nkmq5jgFOGa/builds/gI8SMIQCp406qcgeL/openapi.json