Video to LLM Context – Scenes, Keyframes & Transcript for AI

Turn a video file into context an LLM can use. For every video you get the scene list (scene-cut detection), one keyframe per scene (JPEG link, optional base64), a Whisper transcript aligned to each scene, a keyframe grid image and one ready-to-paste Markdown block (llmContext) for GPT, Claude, Gemini, Llama or your RAG pipeline.

No LLM runs inside the Actor and you need no API key: ffmpeg finds the scenes and Whisper (on CPU) transcribes the speech. You pay $0.002 per keyframe and $0.009 per audio minute. Failed files are free.

What you get for each video

Output	Description
scenes[]	start, end, timecodes, keyframeUrl (JPEG, 768 px wide by default), keyframeBase64 (optional), transcript spoken during that scene
llmContext	Markdown: video facts, then one section per scene (keyframe image link + quote of what is said), then the timestamped full transcript. Also saved as a .md file (llmContextUrl)
gridImageUrl	One contact-sheet JPEG with up to 48 keyframes labelled #scene · mm:ss, so a vision model sees the whole video in one image
transcriptText, language, wordCount	Full Whisper transcript (99 languages, auto-detected)
Metadata	duration, resolution, fps, codec, file size, truncated / truncatedReason

Use cases

• AI agents and RAG: index product demos, webinars, lectures, ads and tutorials as scene-level chunks with timestamps and images.
• Video summaries and chapters: paste llmContext into ChatGPT or Claude and ask for a summary, chapters, a blog post or show notes.
• Ad and creative teardowns: shot list with keyframes and voice-over per shot, ready for a vision model.
• Content moderation and QA: check what is shown and said in each scene of your own uploads.
• Dataset building: scene-aligned image + text pairs from your own footage.

Input

Direct links to video files (MP4, MOV, WebM, MKV, AVI, M4V, TS…), Dropbox / Google Drive / OneDrive file links, an Apify dataset (field with the URL) or an Apify key-value store with video files.

{
  "videoUrls": ["https://cdn.example.com/webinar-2026-10.mp4"],
  "sceneThreshold": 0.3,
  "maxKeyframesPerVideo": 40,
  "maxSceneSeconds": 60,
  "transcribe": true,
  "whisperModel": "base",
  "language": "auto",
  "maxDurationMinutes": 30
}

YouTube, TikTok, Instagram, Facebook, X, Vimeo, Twitch and other platform page links are rejected (free error row): their terms of service do not allow downloading. Use files you own or have the right to process.

Main options:

• Scene-cut sensitivity (sceneThreshold), minimum / maximum scene length (long talking-head scenes are split so there is a keyframe at least every N seconds), max keyframes per video.
• Accurate scene detection: decode every frame for frame-exact cuts (slower, same price). The default fast scan compares the video's own keyframes.
• Whisper model base (default) or tiny, language auto or forced, vocabulary hint.
• Max minutes analyzed per video: you only pay for what is analyzed.

Output example

{
  "type": "video",
  "status": "ok",
  "fileName": "Abbott_and_Costello_Go_to_Mars_Official_Trailer_(1953).webm",
  "durationSec": 51.941,
  "sceneCount": 16,
  "keyframeCount": 16,
  "audioMinutesBilled": 1,
  "language": "en",
  "wordCount": 145,
  "gridImageUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/001-Abbott-and-Costello-Go-to-Mars-Official-grid.jpg",
  "llmContextUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/001-Abbott-and-Costello-Go-to-Mars-Official-context.md",
  "scenes": [
    {
      "index": 2, "start": 3.605, "end": 7.684, "startTimecode": "00:00:03.605",
      "keyframeSec": 5.645,
      "keyframeUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/001-...-scene-002.jpg",
      "transcript": "Today, the United States plan to claim a whole planet through the most extraordinary expedition of our time,"
    }
  ],
  "llmContext": "# Video context: Abbott_and_Costello_Go_to_Mars_Official_Trailer_(1953).webm\n\n- Duration: 00:51 · 628x480 · 25.246 fps\n- Scenes: 16 · Spoken language: en · Words: 145\n\n## Scenes\n\n### Scene 2 · 00:03–00:07\n![Scene 2 keyframe at 00:05](https://...)\n> Today, the United States plan to claim a whole planet ..."
}

Error rows (type: "error", errorCode, error) are never charged.

Pricing

Pay per event, no subscription:

Event	Price	When
Keyframe	$0.002	each keyframe image delivered (one per scene)
Audio minute	$0.009	each started minute of audio transcribed by Whisper

Example: a 10-minute webinar with 30 scenes = 30 × $0.002 + 10 × $0.009 = $0.15. Keyframes only (transcript off): $0.06. Free: rejected links, downloads that fail, files without a video stream, keyframes that cannot be decoded, the grid image, the Markdown file, runs stopped by your maximum cost.

The Actor respects Maximum cost per run: when the budget is not enough for a whole video, only the first part of it is analyzed (truncated: true, truncatedReason: "maxTotalCharge"), and the run stops gracefully.

Actor (Apify Store, Oct 2026)	Price model
This Actor	$0.002 / keyframe + $0.009 / audio minute
kantolabs/video-to-llm-context	$0.002 / keyframe + $0.008 / audio minute
frameprobe/video-scene-splitter	$0.01 / run + $0.02 / video minute
tubetext/video-breakdown-ai	$0.03 / video
ntriqpro/video-intelligence-analyzer	$0.05 / analysis

Use with AI agents / Apify MCP

Add the Actor to your agent through the Apify MCP server (gazidev/video-to-llm-context) or call it from the Apify API, LangChain, LlamaIndex, n8n, Make or Zapier. Give the agent videoUrls, then pass llmContext (text) and gridImageUrl (one image) to the model. For vision models that do not fetch URLs, turn on includeKeyframeBase64.

FAQ

Does it call OpenAI or another LLM? No. It prepares the context; you send it to the model you prefer. No API key needed.

Can I process YouTube or TikTok videos? No. Platform links are refused because their terms forbid downloading. Use your own files (S3, CDN, Dropbox, Drive) or upload them to an Apify key-value store.

How accurate are the scene cuts? The fast scan compares the video's keyframes, so a cut can be off by the keyframe interval (often 1–3 s). Turn on Accurate scene detection for frame-exact cuts.

Long videos? maxDurationMinutes (default 30, up to 240) limits the analyzed part. Scenes longer than maxSceneSeconds are split so there is always a keyframe at least every N seconds.

Which languages? Whisper supports 99 languages and detects the language automatically.

What memory should I use? 2 GB (default) works for most files; 4 GB makes Whisper about twice as fast at the same cost per minute.