# Video to LLM Context – Scenes, Keyframes & Transcript for AI (`gazidev/video-to-llm-context`) Actor

Turn any video file into AI-ready context: scene cuts, one keyframe per scene, a Whisper transcript aligned to each scene, a keyframe grid and a ready-to-paste Markdown block for GPT, Claude or Gemini. MP4, MOV, WebM links, Dropbox, Drive or Apify storage. Failed files free.

- **URL**: https://apify.com/gazidev/video-to-llm-context.md
- **Developed by:** [Cemal Atakli](https://apify.com/gazidev) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 keyframes

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Video to LLM Context – Scenes, Keyframes & Transcript for AI

Turn a video file into **context an LLM can use**. For every video you get the **scene list** (scene-cut detection), **one keyframe per scene** (JPEG link, optional base64), a **Whisper transcript aligned to each scene**, a **keyframe grid** image and one ready-to-paste **Markdown block (`llmContext`)** for GPT, Claude, Gemini, Llama or your RAG pipeline.

No LLM runs inside the Actor and you need no API key: ffmpeg finds the scenes and Whisper (on CPU) transcribes the speech. You pay **$0.002 per keyframe** and **$0.009 per audio minute**. Failed files are free.

### What you get for each video

| Output | Description |
|---|---|
| `scenes[]` | `start`, `end`, timecodes, `keyframeUrl` (JPEG, 768 px wide by default), `keyframeBase64` (optional), `transcript` spoken during that scene |
| `llmContext` | Markdown: video facts, then one section per scene (keyframe image link + quote of what is said), then the timestamped full transcript. Also saved as a `.md` file (`llmContextUrl`) |
| `gridImageUrl` | One contact-sheet JPEG with up to 48 keyframes labelled `#scene · mm:ss`, so a vision model sees the whole video in **one image** |
| `transcriptText`, `language`, `wordCount` | Full Whisper transcript (99 languages, auto-detected) |
| Metadata | duration, resolution, fps, codec, file size, `truncated` / `truncatedReason` |

### Use cases

- **AI agents and RAG**: index product demos, webinars, lectures, ads and tutorials as scene-level chunks with timestamps and images.
- **Video summaries and chapters**: paste `llmContext` into ChatGPT or Claude and ask for a summary, chapters, a blog post or show notes.
- **Ad and creative teardowns**: shot list with keyframes and voice-over per shot, ready for a vision model.
- **Content moderation and QA**: check what is shown and said in each scene of your own uploads.
- **Dataset building**: scene-aligned image + text pairs from your own footage.

### Input

Direct links to video files (MP4, MOV, WebM, MKV, AVI, M4V, TS…), Dropbox / Google Drive / OneDrive file links, an **Apify dataset** (field with the URL) or an **Apify key-value store** with video files.

```json
{
  "videoUrls": ["https://cdn.example.com/webinar-2026-10.mp4"],
  "sceneThreshold": 0.3,
  "maxKeyframesPerVideo": 40,
  "maxSceneSeconds": 60,
  "transcribe": true,
  "whisperModel": "base",
  "language": "auto",
  "maxDurationMinutes": 30
}
```

**YouTube, TikTok, Instagram, Facebook, X, Vimeo, Twitch and other platform page links are rejected** (free error row): their terms of service do not allow downloading. Use files you own or have the right to process.

Main options:

- **Scene-cut sensitivity** (`sceneThreshold`), **minimum / maximum scene length** (long talking-head scenes are split so there is a keyframe at least every N seconds), **max keyframes per video**.
- **Accurate scene detection**: decode every frame for frame-exact cuts (slower, same price). The default fast scan compares the video's own keyframes.
- **Whisper model** `base` (default) or `tiny`, language auto or forced, vocabulary hint.
- **Max minutes analyzed per video**: you only pay for what is analyzed.

### Output example

```json
{
  "type": "video",
  "status": "ok",
  "fileName": "Abbott_and_Costello_Go_to_Mars_Official_Trailer_(1953).webm",
  "durationSec": 51.941,
  "sceneCount": 16,
  "keyframeCount": 16,
  "audioMinutesBilled": 1,
  "language": "en",
  "wordCount": 145,
  "gridImageUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/001-Abbott-and-Costello-Go-to-Mars-Official-grid.jpg",
  "llmContextUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/001-Abbott-and-Costello-Go-to-Mars-Official-context.md",
  "scenes": [
    {
      "index": 2, "start": 3.605, "end": 7.684, "startTimecode": "00:00:03.605",
      "keyframeSec": 5.645,
      "keyframeUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/001-...-scene-002.jpg",
      "transcript": "Today, the United States plan to claim a whole planet through the most extraordinary expedition of our time,"
    }
  ],
  "llmContext": "# Video context: Abbott_and_Costello_Go_to_Mars_Official_Trailer_(1953).webm\n\n- Duration: 00:51 · 628x480 · 25.246 fps\n- Scenes: 16 · Spoken language: en · Words: 145\n\n## Scenes\n\n### Scene 2 · 00:03–00:07\n![Scene 2 keyframe at 00:05](https://...)\n> Today, the United States plan to claim a whole planet ..."
}
```

Error rows (`type: "error"`, `errorCode`, `error`) are never charged.

### Pricing

Pay per event, no subscription:

| Event | Price | When |
|---|---|---|
| Keyframe | **$0.002** | each keyframe image delivered (one per scene) |
| Audio minute | **$0.009** | each started minute of audio transcribed by Whisper |

Example: a 10-minute webinar with 30 scenes = 30 × $0.002 + 10 × $0.009 = **$0.15**. Keyframes only (transcript off): $0.06.
Free: rejected links, downloads that fail, files without a video stream, keyframes that cannot be decoded, the grid image, the Markdown file, runs stopped by your maximum cost.

The Actor respects **Maximum cost per run**: when the budget is not enough for a whole video, only the first part of it is analyzed (`truncated: true`, `truncatedReason: "maxTotalCharge"`), and the run stops gracefully.

| Actor (Apify Store, Oct 2026) | Price model |
|---|---|
| **This Actor** | $0.002 / keyframe + $0.009 / audio minute |
| kantolabs/video-to-llm-context | $0.002 / keyframe + $0.008 / audio minute |
| frameprobe/video-scene-splitter | $0.01 / run + $0.02 / video minute |
| tubetext/video-breakdown-ai | $0.03 / video |
| ntriqpro/video-intelligence-analyzer | $0.05 / analysis |

### Use with AI agents / Apify MCP

Add the Actor to your agent through the [Apify MCP server](https://mcp.apify.com) (`gazidev/video-to-llm-context`) or call it from the Apify API, LangChain, LlamaIndex, n8n, Make or Zapier. Give the agent `videoUrls`, then pass `llmContext` (text) and `gridImageUrl` (one image) to the model. For vision models that do not fetch URLs, turn on `includeKeyframeBase64`.

### FAQ

**Does it call OpenAI or another LLM?** No. It prepares the context; you send it to the model you prefer. No API key needed.

**Can I process YouTube or TikTok videos?** No. Platform links are refused because their terms forbid downloading. Use your own files (S3, CDN, Dropbox, Drive) or upload them to an Apify key-value store.

**How accurate are the scene cuts?** The fast scan compares the video's keyframes, so a cut can be off by the keyframe interval (often 1–3 s). Turn on *Accurate scene detection* for frame-exact cuts.

**Long videos?** `maxDurationMinutes` (default 30, up to 240) limits the analyzed part. Scenes longer than `maxSceneSeconds` are split so there is always a keyframe at least every N seconds.

**Which languages?** Whisper supports 99 languages and detects the language automatically.

**What memory should I use?** 2 GB (default) works for most files; 4 GB makes Whisper about twice as fast at the same cost per minute.

# Actor input Schema

## `videoUrls` (type: `array`):

Direct links to video files: MP4, MOV, WebM, MKV, AVI, M4V, TS and more (your CDN, S3/R2/GCS bucket, Dropbox or Google Drive file shared as 'Anyone with the link', OneDrive). **YouTube, TikTok, Instagram, Facebook, X, Vimeo, Twitch and other platform page links are rejected** (free error row): their terms of service forbid downloading. Only submit videos you have the right to process.

## `datasetId` (type: `string`):

Optional. Chain after another Actor: every value of the field below (string or array of strings) is processed like a video URL.

## `datasetField` (type: `string`):

Name of the dataset field that holds the video file URL, e.g. `url`, `videoUrl` or `fileUrl`.

## `keyValueStoreId` (type: `string`):

Optional. Every record in this store is analyzed as a video file, e.g. files you uploaded with the Apify API or another Actor's output. JSON/text records become free error rows.

## `keyValueStorePrefix` (type: `string`):

Optional key prefix filter for the key-value store (e.g. `video-`).

## `maxVideos` (type: `integer`):

Stop after this many inputs (0 = no limit). Useful when reading a large dataset or store.

## `sceneThreshold` (type: `number`):

ffmpeg scene-change score (0.05–0.95) above which a new scene starts. Lower = more scenes (0.2 for subtle cuts), higher = only hard cuts (0.4–0.5).

## `minSceneSeconds` (type: `number`):

Cuts closer than this to the previous one are merged, so flashes and fast montages do not produce dozens of near-identical keyframes.

## `maxSceneSeconds` (type: `number`):

Longer scenes (talking heads, screen recordings, lectures) are split evenly so the context still gets a keyframe at least this often. 0 = never split.

## `maxKeyframesPerVideo` (type: `integer`):

Upper limit on scenes/keyframes per video (one keyframe per scene, each charged). If more cuts are found, evenly spread cut points are kept.

## `accurateScenes` (type: `boolean`):

Off (default): fast scan that compares the video's own keyframes - cut times can be off by a second or two on files with sparse keyframes. On: every frame is decoded for frame-exact cuts (several times slower on HD video, same price).

## `keyframeWidth` (type: `integer`):

Keyframes are scaled down to this width (never up). 768 px is a good size for vision LLMs (GPT-4o, Claude, Gemini) and keeps image tokens low.

## `keyframeQuality` (type: `integer`):

JPEG quality of the keyframes (30–100).

## `includeKeyframeBase64` (type: `boolean`):

Add `keyframeBase64` (JPEG, base64) to every scene so you can send images to a vision model without downloading the links. Makes rows much larger; dropped automatically if a row would exceed 8 MB.

## `keyframeGrid` (type: `boolean`):

Save one contact-sheet JPEG with up to 48 keyframes, labelled with scene number and timecode (`gridImageUrl`). One image = one vision-model call for the whole video. Free.

## `transcribe` (type: `boolean`):

Transcribe the audio track with Whisper (on-CPU, no API key) and align the text to the scenes. Charged per started audio minute. Turn off for keyframes only.

## `whisperModel` (type: `string`):

`base`: good accuracy for clear speech in 99 languages. `tiny`: about 2× faster, fine for clean single-speaker English. Same price.

## `language` (type: `string`):

`auto` detects the language from the first 30 seconds of speech. Or force a Whisper code (`en`, `es`, `de`, `fr`, `pt`, `it`, `nl`, `tr`, `ru`, `ja`, `zh`, `ar`, `hi`…) for short or noisy clips.

## `vocabularyHint` (type: `string`):

Optional names and jargon that help Whisper spell them right, e.g. `Apify, Kubernetes, Dr. Nguyen`.

## `includeSegments` (type: `boolean`):

Add the raw Whisper `segments` (start, end, text) to each row. The per-scene transcript and the Markdown already contain the text.

## `imageLinksInContext` (type: `boolean`):

Put a Markdown image link for every keyframe (and the grid) into `llmContext`. Turn off for a text-only context block.

## `maxDurationMinutes` (type: `number`):

Only the first N minutes of each video are analyzed (scenes and transcript, `truncated: true`). You pay only for what is analyzed.

## `maxFileSizeMB` (type: `integer`):

Larger downloads are stopped (free error row). 1 hour of 720p MP4 is about 500–1500 MB.

## `downloadTimeoutSecs` (type: `integer`):

Maximum time to download one file. Downloads are retried on network errors and HTTP 429/5xx.

## Actor input object example

```json
{
  "videoUrls": [
    "https://upload.wikimedia.org/wikipedia/commons/2/2c/Abbott_and_Costello_Go_to_Mars_Official_Trailer_%281953%29.webm"
  ],
  "datasetField": "url",
  "maxVideos": 0,
  "sceneThreshold": 0.3,
  "minSceneSeconds": 2,
  "maxSceneSeconds": 60,
  "maxKeyframesPerVideo": 40,
  "accurateScenes": false,
  "keyframeWidth": 768,
  "keyframeQuality": 80,
  "includeKeyframeBase64": false,
  "keyframeGrid": true,
  "transcribe": true,
  "whisperModel": "base",
  "language": "auto",
  "includeSegments": false,
  "imageLinksInContext": true,
  "maxDurationMinutes": 30,
  "maxFileSizeMB": 2000,
  "downloadTimeoutSecs": 900
}
```

# Actor output Schema

## `videos` (type: `string`):

No description

## `llmContext` (type: `string`):

No description

## `errors` (type: `string`):

No description

## `files` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "videoUrls": [
        "https://upload.wikimedia.org/wikipedia/commons/2/2c/Abbott_and_Costello_Go_to_Mars_Official_Trailer_%281953%29.webm"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("gazidev/video-to-llm-context").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "videoUrls": ["https://upload.wikimedia.org/wikipedia/commons/2/2c/Abbott_and_Costello_Go_to_Mars_Official_Trailer_%281953%29.webm"] }

# Run the Actor and wait for it to finish
run = client.actor("gazidev/video-to-llm-context").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "videoUrls": [
    "https://upload.wikimedia.org/wikipedia/commons/2/2c/Abbott_and_Costello_Go_to_Mars_Official_Trailer_%281953%29.webm"
  ]
}' |
apify call gazidev/video-to-llm-context --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,gazidev/video-to-llm-context"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/3hBOkKlwW42aZwk3D/builds/jwOvE51z566JIfxIR/openapi.json