Video to LLM Context - Keyframes, Transcript & Scenes for AI Agents

Video to LLM Context turns any video file into context a language model can actually read: a scene list, one keyframe image per scene, the speech transcript aligned to each scene, an optional grid image of all keyframes, and a ready-to-paste llmContext text block. One API call, one JSON row per video.

GPT, Claude, Gemini and most AI agents cannot watch a video - they read images and text. The standard recipe in every "summarize a video with AI" guide is sample frames + transcribe the audio. This Actor does both in one step and lines them up, so your own model sees what is on screen and what is said in each scene. There is no LLM inside and no API key to bring: you keep full control of the model and the prompt.

What does Video to LLM Context do?

For every direct video link you give it, the Actor:

1. Downloads the file and reads its metadata: duration, resolution, fps, codecs, whether it has audio.
2. Finds the scene cuts with ffmpeg and turns them into at most N scenes. The strongest cuts win, spread over the whole video; long shots (talking heads, screen recordings) are split into parts at pauses in the speech, so a sentence is not cut in half.
3. Saves one keyframe per scene (from the middle of the scene, stepping past black frames) as JPG/PNG/WEBP with a public link.
4. Transcribes the speech with OpenAI Whisper (99 languages, auto-detect) and assigns every word to the scene it was spoken in. Each scene gets its own transcript and timestamped segments.
5. Tiles all keyframes into one labelled grid image (free) - send one image to a vision model instead of 12.
6. Writes llmContext: plain text with one block per scene (time range, image link, speech) that you paste straight into a prompt.

Use cases

• AI agents and MCP tools - give Claude, ChatGPT or a custom agent the ability to "watch" a video file.
• Summaries, chapters and show notes from webinars, lectures, demos and meetings.
• Video RAG - index each scene (image + speech) as its own chunk for search over video libraries.
• n8n / Make / Zapier flows - "new video in a bucket -> context -> GPT summary -> Slack".
• Content moderation and tagging - check what is shown and said, scene by scene.
• Ad and creative analysis of your own or licensed videos: hook, scenes, voice-over per shot.

How to use it

1. Click Try for free.
2. Paste direct links to your video files into Video file URLs (S3/R2/GCS, a CDN, Dropbox or Google Drive share links, an Apify key-value store...).
3. Optionally set Max scenes / keyframes per video (default 12) and the Spoken language.
4. Click Start. Copy llmContext into your prompt, or attach the keyframe / grid images to a vision model.

Platform pages (YouTube, TikTok, Instagram, Facebook, X, Vimeo...) are refused on purpose: their terms forbid downloading. Use files you own or are allowed to process.

Input

Field	What it does	Default
videoUrls	Direct links to video files (MP4, MOV, WEBM, MKV, AVI, MPEG-TS...)	a 21 s public-domain NASA clip
maxKeyframes	Most scenes (= keyframes) per video, 1-100	12
transcribe	Transcribe speech and align it to the scenes	true
language	auto or a 2-letter code (en, es, de, ja...)	auto
sceneThreshold	Cut sensitivity (lower = more cuts)	0.3
minSceneSeconds / maxSceneSeconds	Merge cuts closer than this / split shots longer than this	2 / 60
gridImage	One labelled image with every keyframe (free)	true
includeImageUrlsInContext	Put image links into llmContext	true
keyframeMaxWidth, imageFormat, imageQuality	Keyframe size and format	1024, jpg, 85
skipBlankFrames	Step past black / single-colour frames (blank ones are not charged)	true
maxDurationMinutes, maxVideoSizeMb, downloadTimeoutSecs	Limits (over-limit videos are free)	120, 1000, 600

Output

One dataset item per video. Real output of the default input from a cloud run (signatures shortened, scene 3 omitted):

{
  "fileName": "sample-vertical-talk.mp4",
  "durationSeconds": 21.55,
  "width": 720,
  "height": 1280,
  "fps": 29.97,
  "hasAudio": true,
  "videoCodec": "h264",
  "status": "ok",
  "sceneCount": 3,
  "cutsDetected": 2,
  "keyframeCount": 3,
  "gridImageUrl": "https://api.apify.com/v2/key-value-stores/vmP2lh2IKb71APELa/records/d1b071e0-sample-vertical-talk-keyframe-grid.jpg?signature=...",
  "transcriptLanguage": "en",
  "billedKeyframes": 3,
  "billedAudioMinutes": 1,
  "scenes": [
    {
      "index": 1,
      "start": 0,
      "end": 2.936,
      "boundary": "start",
      "keyframeUrl": "https://api.apify.com/v2/key-value-stores/vmP2lh2IKb71APELa/records/d1b071e0-sample-vertical-talk-scene001-000001468ms.jpg?signature=...",
      "transcript": "rock and ice that we call it debris disc.",
      "segments": [{ "start": 0, "end": 1.86, "text": "rock and ice that we call it debris disc." }]
    },
    {
      "index": 2,
      "start": 2.936,
      "end": 13.246,
      "boundary": "cut",
      "cutScore": 0.7773,
      "keyframeUrl": "https://api.apify.com/v2/key-value-stores/vmP2lh2IKb71APELa/records/d1b071e0-sample-vertical-talk-scene002-000008090ms.jpg?signature=...",
      "transcript": "Inside that disc is essential clearing in the larger plant test moles. And inside that central clearing, it's a planet more massive than any in our solar system.",
      "segments": [
        { "start": 2.94, "end": 5.29, "text": "Inside that disc is essential clearing" },
        { "start": 5.69, "end": 7.15, "text": "in the larger plant test moles." }
      ]
    }
  ],
  "transcript": { "status": "ok", "language": "en", "languageProbability": 0.999, "model": "Whisper base", "transcribedSeconds": 21.504, "wordCount": 68 },
  "transcriptText": "rock and ice that we call it debris disc. Inside that disc is essential clearing ...",
  "error": null
}

Each scene's keyframe object also carries the exact timestamp, timecode, width, height and storage key. The llmContext field of the same run (image links shortened):

VIDEO: sample-vertical-talk.mp4
duration 0:21 | 720x1280 | 29.97 fps | speech language en
3 scenes; one keyframe image per scene (image links below; scene numbers match the order of the images).
All keyframes in one grid image (labelled with scene number and time): https://...keyframe-grid.jpg

[Scene 1] 0:00-0:02 (keyframe at 0:01)
Image: https://...scene001-000001468ms.jpg
Speech: rock and ice that we call it debris disc.

[Scene 2] 0:02-0:13 (keyframe at 0:08)
Image: https://...scene002-000008090ms.jpg
Speech: Inside that disc is essential clearing in the larger plant test moles. And inside that central clearing, it's a planet more massive than any in our solar system.

All image links open without an Apify login, so a vision model or an agent can fetch them directly. A video that could not be processed gets a row with "status": "failed" and an error that says how to fix it - and costs nothing.

How much does it cost to turn a video into LLM context?

Pay per event, no monthly fee, no per-video fee:

Event	Price	When
Keyframe extracted	$0.002 per keyframe ($2 per 1,000)	one per scene image delivered
Audio minute transcribed	$0.008 per started minute	only when speech was found and transcribed

The grid image, metadata, scene list and llmContext are free. Free rows: refused links, failed downloads, unreadable files, over-limit videos. Videos with no audio track or no speech pay for keyframes only.

Worked examples:

• A 10-minute webinar, 12 scenes: 12 x $0.002 + 10 x $0.008 = $0.104
• A 30-second ad, 8 scenes: 8 x $0.002 + 1 x $0.008 = $0.024
• A 5-minute silent screen recording, 12 keyframes, no speech = $0.024

Higher Apify plans get lower prices automatically (Silver -10 %, Gold -20 %). Set Maximum cost per run and the Actor plans within it: fewer keyframes or a shorter transcript for the last video, then free skipped rows.

Speed (measured on Apify, October 2026, default 2 GB)

Test video	Length	Run time
Default sample (vertical talk clip)	21 s	17 s
Edited science podcast episode, 68 cuts	5 min 55 s	2.5 min
Single-camera interview	9 min	3.8 min

Most of the time is speech recognition (about 0.3x real time). Keyframes-only runs (transcribe: false) take seconds.

Integrations and API

Run it from the Apify API, the Python or JavaScript client, schedules, webhooks, n8n, Make, Zapier, or as an MCP tool for AI agents through the Apify MCP server. The input schema doubles as the tool's parameters, so an agent only has to pass videoUrls.

Related Kanto Labs Actors

• Video Frame Extractor - more control over frames: every N seconds, fixed counts, scene changes only, contact sheets.
• Transcribe Audio & Video to Text - transcripts with SRT/VTT subtitle files and word timings, also for audio-only files (MP3, WAV, M4A...).

FAQ

Does this Actor call ChatGPT, Claude or another LLM?

No. It prepares the video for your model. Nothing is sent to a third-party AI service, and you choose the model and the prompt.

Which videos work?

Any file ffmpeg can decode (MP4, MOV, WEBM, MKV, AVI, MPEG-TS...), up to 300 minutes and 4 GB. The link must point to the file itself. Dropbox and Google Drive share links are converted to direct downloads automatically.

Can I use YouTube or TikTok links?

No. Those platforms forbid downloading in their terms, so the Actor refuses them (free row).

How accurate is the transcript?

It uses Whisper base: good on clear speech, weaker on names, heavy accents and noisy audio. Set language when you know it - that helps short clips most.

What if the video has music but no speech?

Then there is no transcript and no audio charge; you still get the scenes and keyframes.

How many keyframes should I ask for?

8-16 works for most prompts. The grid image lets you send them all as one picture to save vision tokens.

Where are the images stored?

In the run's key-value store on Apify, with public signed links. Download them if you need them beyond your plan's data retention.

Something failed - what now?

Read the error column: it says what went wrong and how to fix it. Still stuck? Open an issue in the Issues tab with the video link and we will look at it.