# Video to LLM Context - Keyframes, Transcript & Scenes for AI (`kantolabs/video-to-llm-context`) Actor Turn any video file into context an AI agent can read: scene list, one keyframe image per scene, Whisper transcript aligned to each scene, a keyframe grid and a ready-to-paste text block. No LLM inside. $0.002 per keyframe + $0.008 per audio minute. - **URL**: https://apify.com/kantolabs/video-to-llm-context.md - **Developed by:** [Kanto Labs](https://apify.com/kantolabs) (community) - **Categories:** Videos, AI, Developer tools - **Stats:** 2 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.60 / 1,000 keyframes This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Video to LLM Context - Keyframes, Transcript & Scenes for AI Agents **Video to LLM Context** turns any video file into context a language model can actually read: a **scene list**, **one keyframe image per scene**, the **speech transcript aligned to each scene**, an optional **grid image** of all keyframes, and a ready-to-paste **`llmContext`** text block. One API call, one JSON row per video. GPT, Claude, Gemini and most AI agents cannot watch a video - they read images and text. The standard recipe in every "summarize a video with AI" guide is *sample frames + transcribe the audio*. This Actor does both in one step and lines them up, so your own model sees what is on screen **and** what is said in each scene. There is no LLM inside and no API key to bring: you keep full control of the model and the prompt. ### What does Video to LLM Context do? For every direct video link you give it, the Actor: 1. Downloads the file and reads its **metadata**: duration, resolution, fps, codecs, whether it has audio. 2. Finds the **scene cuts** with ffmpeg and turns them into at most *N* scenes. The strongest cuts win, spread over the whole video; long shots (talking heads, screen recordings) are split into parts at **pauses in the speech**, so a sentence is not cut in half. 3. Saves **one keyframe per scene** (from the middle of the scene, stepping past black frames) as JPG/PNG/WEBP with a public link. 4. Transcribes the speech with **OpenAI Whisper** (99 languages, auto-detect) and assigns every word to the scene it was spoken in. Each scene gets its own `transcript` and timestamped `segments`. 5. Tiles all keyframes into **one labelled grid image** (free) - send one image to a vision model instead of 12. 6. Writes **`llmContext`**: plain text with one block per scene (time range, image link, speech) that you paste straight into a prompt. ### Use cases - **AI agents and MCP tools** - give Claude, ChatGPT or a custom agent the ability to "watch" a video file. - **Summaries, chapters and show notes** from webinars, lectures, demos and meetings. - **Video RAG** - index each scene (image + speech) as its own chunk for search over video libraries. - **n8n / Make / Zapier flows** - "new video in a bucket -> context -> GPT summary -> Slack". - **Content moderation and tagging** - check what is shown and said, scene by scene. - **Ad and creative analysis** of your own or licensed videos: hook, scenes, voice-over per shot. ### How to use it 1. Click **Try for free**. 2. Paste direct links to your video files into **Video file URLs** (S3/R2/GCS, a CDN, Dropbox or Google Drive share links, an Apify key-value store...). 3. Optionally set **Max scenes / keyframes per video** (default 12) and the **Spoken language**. 4. Click **Start**. Copy `llmContext` into your prompt, or attach the keyframe / grid images to a vision model. Platform pages (YouTube, TikTok, Instagram, Facebook, X, Vimeo...) are refused on purpose: their terms forbid downloading. Use files you own or are allowed to process. ### Input | Field | What it does | Default | |---|---|---| | `videoUrls` | Direct links to video files (MP4, MOV, WEBM, MKV, AVI, MPEG-TS...) | a 21 s public-domain NASA clip | | `maxKeyframes` | Most scenes (= keyframes) per video, 1-100 | `12` | | `transcribe` | Transcribe speech and align it to the scenes | `true` | | `language` | `auto` or a 2-letter code (`en`, `es`, `de`, `ja`...) | `auto` | | `sceneThreshold` | Cut sensitivity (lower = more cuts) | `0.3` | | `minSceneSeconds` / `maxSceneSeconds` | Merge cuts closer than this / split shots longer than this | `2` / `60` | | `gridImage` | One labelled image with every keyframe (free) | `true` | | `includeImageUrlsInContext` | Put image links into `llmContext` | `true` | | `keyframeMaxWidth`, `imageFormat`, `imageQuality` | Keyframe size and format | `1024`, `jpg`, `85` | | `skipBlankFrames` | Step past black / single-colour frames (blank ones are not charged) | `true` | | `maxDurationMinutes`, `maxVideoSizeMb`, `downloadTimeoutSecs` | Limits (over-limit videos are free) | `120`, `1000`, `600` | ### Output One dataset item **per video**. Real output of the default input from a cloud run (signatures shortened, scene 3 omitted): ```json { "fileName": "sample-vertical-talk.mp4", "durationSeconds": 21.55, "width": 720, "height": 1280, "fps": 29.97, "hasAudio": true, "videoCodec": "h264", "status": "ok", "sceneCount": 3, "cutsDetected": 2, "keyframeCount": 3, "gridImageUrl": "https://api.apify.com/v2/key-value-stores/vmP2lh2IKb71APELa/records/d1b071e0-sample-vertical-talk-keyframe-grid.jpg?signature=...", "transcriptLanguage": "en", "billedKeyframes": 3, "billedAudioMinutes": 1, "scenes": [ { "index": 1, "start": 0, "end": 2.936, "boundary": "start", "keyframeUrl": "https://api.apify.com/v2/key-value-stores/vmP2lh2IKb71APELa/records/d1b071e0-sample-vertical-talk-scene001-000001468ms.jpg?signature=...", "transcript": "rock and ice that we call it debris disc.", "segments": [{ "start": 0, "end": 1.86, "text": "rock and ice that we call it debris disc." }] }, { "index": 2, "start": 2.936, "end": 13.246, "boundary": "cut", "cutScore": 0.7773, "keyframeUrl": "https://api.apify.com/v2/key-value-stores/vmP2lh2IKb71APELa/records/d1b071e0-sample-vertical-talk-scene002-000008090ms.jpg?signature=...", "transcript": "Inside that disc is essential clearing in the larger plant test moles. And inside that central clearing, it's a planet more massive than any in our solar system.", "segments": [ { "start": 2.94, "end": 5.29, "text": "Inside that disc is essential clearing" }, { "start": 5.69, "end": 7.15, "text": "in the larger plant test moles." } ] } ], "transcript": { "status": "ok", "language": "en", "languageProbability": 0.999, "model": "Whisper base", "transcribedSeconds": 21.504, "wordCount": 68 }, "transcriptText": "rock and ice that we call it debris disc. Inside that disc is essential clearing ...", "error": null } ``` Each scene's `keyframe` object also carries the exact timestamp, timecode, width, height and storage key. The `llmContext` field of the same run (image links shortened): ```text VIDEO: sample-vertical-talk.mp4 duration 0:21 | 720x1280 | 29.97 fps | speech language en 3 scenes; one keyframe image per scene (image links below; scene numbers match the order of the images). All keyframes in one grid image (labelled with scene number and time): https://...keyframe-grid.jpg [Scene 1] 0:00-0:02 (keyframe at 0:01) Image: https://...scene001-000001468ms.jpg Speech: rock and ice that we call it debris disc. [Scene 2] 0:02-0:13 (keyframe at 0:08) Image: https://...scene002-000008090ms.jpg Speech: Inside that disc is essential clearing in the larger plant test moles. And inside that central clearing, it's a planet more massive than any in our solar system. ``` All image links open without an Apify login, so a vision model or an agent can fetch them directly. A video that could not be processed gets a row with `"status": "failed"` and an `error` that says how to fix it - and costs nothing. ### How much does it cost to turn a video into LLM context? Pay per event, **no monthly fee**, no per-video fee: | Event | Price | When | |---|---|---| | Keyframe extracted | **$0.002** per keyframe ($2 per 1,000) | one per scene image delivered | | Audio minute transcribed | **$0.008** per started minute | only when speech was found and transcribed | The grid image, metadata, scene list and `llmContext` are free. **Free rows:** refused links, failed downloads, unreadable files, over-limit videos. Videos with no audio track or no speech pay for keyframes only. Worked examples: - A 10-minute webinar, 12 scenes: 12 x $0.002 + 10 x $0.008 = **$0.104** - A 30-second ad, 8 scenes: 8 x $0.002 + 1 x $0.008 = **$0.024** - A 5-minute silent screen recording, 12 keyframes, no speech = **$0.024** Higher Apify plans get lower prices automatically (Silver -10 %, Gold -20 %). Set **Maximum cost per run** and the Actor plans within it: fewer keyframes or a shorter transcript for the last video, then free `skipped` rows. ### Speed (measured on Apify, October 2026, default 2 GB) | Test video | Length | Run time | |---|---|---| | Default sample (vertical talk clip) | 21 s | 17 s | | Edited science podcast episode, 68 cuts | 5 min 55 s | 2.5 min | | Single-camera interview | 9 min | 3.8 min | Most of the time is speech recognition (about 0.3x real time). Keyframes-only runs (`transcribe: false`) take seconds. ### Integrations and API Run it from the Apify API, the Python or JavaScript client, schedules, webhooks, **n8n, Make, Zapier**, or as an **MCP tool** for AI agents through the Apify MCP server. The input schema doubles as the tool's parameters, so an agent only has to pass `videoUrls`. ### Related Kanto Labs Actors - [Video Frame Extractor](https://apify.com/kantolabs/video-frame-extractor) - more control over frames: every N seconds, fixed counts, scene changes only, contact sheets. - [Transcribe Audio & Video to Text](https://apify.com/kantolabs/audio-video-to-text) - transcripts with SRT/VTT subtitle files and word timings, also for audio-only files (MP3, WAV, M4A...). ### FAQ #### Does this Actor call ChatGPT, Claude or another LLM? No. It prepares the video for *your* model. Nothing is sent to a third-party AI service, and you choose the model and the prompt. #### Which videos work? Any file ffmpeg can decode (MP4, MOV, WEBM, MKV, AVI, MPEG-TS...), up to 300 minutes and 4 GB. The link must point to the file itself. Dropbox and Google Drive share links are converted to direct downloads automatically. #### Can I use YouTube or TikTok links? No. Those platforms forbid downloading in their terms, so the Actor refuses them (free row). #### How accurate is the transcript? It uses Whisper base: good on clear speech, weaker on names, heavy accents and noisy audio. Set `language` when you know it - that helps short clips most. #### What if the video has music but no speech? Then there is no transcript and no audio charge; you still get the scenes and keyframes. #### How many keyframes should I ask for? 8-16 works for most prompts. The grid image lets you send them all as one picture to save vision tokens. #### Where are the images stored? In the run's key-value store on Apify, with public signed links. Download them if you need them beyond your plan's data retention. #### Something failed - what now? Read the `error` column: it says what went wrong and how to fix it. Still stuck? Open an issue in the Issues tab with the video link and we will look at it. # Actor input Schema ## `videoUrls` (type: `array`): Direct links to video files (MP4, MOV, WEBM, MKV, AVI, M4V, MPEG-TS...), one per line. Your own storage, S3/R2/GCS (public or pre-signed URLs), CDNs, an Apify key-value store, Dropbox and Google Drive share links (shared as 'anyone with the link') all work. Pages of platforms such as YouTube, TikTok or Instagram are refused. ## `maxKeyframes` (type: `integer`): Upper limit of scenes per video; each scene gets one keyframe image ($0.002 each). The strongest cuts are kept first. 8-16 suits most AI prompts; long videos without cuts are split into parts of at most 'Max scene length'. ## `transcribe` (type: `boolean`): Transcribe the audio track with Whisper and align the words to each scene ($0.008 per started audio minute). Videos without speech or audio are not charged for audio. Turn off for keyframes only. ## `language` (type: `string`): 'auto' detects the language. Or a 2-letter ISO 639-1 code (en, es, de, fr, pt, ja, zh...) to skip detection - recommended for short clips. ## `sceneThreshold` (type: `string`): Minimum ffmpeg scene score (0-1) that counts as a cut. Lower finds subtler changes. ## `minSceneSeconds` (type: `integer`): Cuts closer together than this are merged, so fast montages do not produce a flood of tiny scenes. ## `maxSceneSeconds` (type: `integer`): Shots longer than this (talking heads, screen recordings) are split into equal parts at pauses in the speech, while the scene limit allows. Real cuts are never moved. ## `gridImage` (type: `boolean`): Also save ONE image with every keyframe tiled and labelled (scene number + time) - send a single image to a vision model instead of many. Not charged. ## `includeImageUrlsInContext` (type: `boolean`): Put the keyframe and grid image links into the plain-text llmContext field. Turn off for a shorter text block (links stay in the scenes list). ## `keyframeMaxWidth` (type: `integer`): Keyframes wider than this are scaled down (aspect kept). 1024 is a good size for vision models. 0 = original size. ## `imageFormat` (type: `string`): Format of the keyframe images. ## `imageQuality` (type: `integer`): 30-100. ## `skipBlankFrames` (type: `boolean`): When a keyframe is black or one flat colour, step forward up to 2 seconds inside the scene to a frame with content; if none, that scene has no image and it is not charged. ## `maxDurationMinutes` (type: `integer`): Longer videos are skipped (free). Up to 300. ## `maxVideoSizeMb` (type: `integer`): Larger files are skipped (free). ## `downloadTimeoutSecs` (type: `integer`): Give up on a video that does not download within this time (free). ## Actor input object example ```json { "videoUrls": [ "https://api.apify.com/v2/key-value-stores/aXcXwfChlioPWAUJv/records/sample-vertical-talk.mp4?signature=105QVevs72w3z5gMQzgIJ" ], "maxKeyframes": 12, "transcribe": true, "language": "auto", "sceneThreshold": "0.3", "minSceneSeconds": 2, "maxSceneSeconds": 60, "gridImage": true, "includeImageUrlsInContext": true, "keyframeMaxWidth": 1024, "imageFormat": "jpg", "imageQuality": 85, "skipBlankFrames": true, "maxDurationMinutes": 120, "maxVideoSizeMb": 1000, "downloadTimeoutSecs": 600 } ``` # Actor output Schema ## `results` (type: `string`): No description ## `files` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "videoUrls": [ "https://api.apify.com/v2/key-value-stores/aXcXwfChlioPWAUJv/records/sample-vertical-talk.mp4?signature=105QVevs72w3z5gMQzgIJ" ] }; // Run the Actor and wait for it to finish const run = await client.actor("kantolabs/video-to-llm-context").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "videoUrls": ["https://api.apify.com/v2/key-value-stores/aXcXwfChlioPWAUJv/records/sample-vertical-talk.mp4?signature=105QVevs72w3z5gMQzgIJ"] } # Run the Actor and wait for it to finish run = client.actor("kantolabs/video-to-llm-context").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "videoUrls": [ "https://api.apify.com/v2/key-value-stores/aXcXwfChlioPWAUJv/records/sample-vertical-talk.mp4?signature=105QVevs72w3z5gMQzgIJ" ] }' | apify call kantolabs/video-to-llm-context --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,kantolabs/video-to-llm-context" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/KzmEfQVKbpCzJhbGq/builds/nNXnvtkuch2gaRODR/openapi.json