Content Based Video Narration of Gameplay with Vision Language Models
| Source: arXiv AI
Tags: vision-language models, video narration, gaming AI, TTS, multimodal, on-device AI
A no-training system turns any gameplay recording into esports-style commentary using a VLM and TTS, with a 3x3 frame mosaic trick that cuts image payload 9x and context-conditioned prompting to suppress repetitive narration — with a fully local TTS option on Apple silicon.
Details
Live game commentary exists almost exclusively for professional esports broadcasts. This paper presents a content-based video narration pipeline that runs on arbitrary gameplay recordings using a general-purpose vision-language model (VLM) and text-to-speech back end, requiring no game-specific instrumentation, no engine telemetry, and no task-specific training. Three technical mechanisms carry the system. Temporal mosaic packing arranges nine uniformly sampled frames from a segment into a single 3x3 image — letting an image-native VLM reason about motion while consuming one image payload per segment instead of nine, a 9x reduction in API cost for image-heavy workflows. Context-conditioned prompting replays the K most recent narrations as assistant-role history, which suppresses the repetition that plagues per-segment captioning when scenes change slowly. Duration-conditioned generation plus elastic alignment constrains narration length in the prompt, then time-scales or pads synthesized audio to fill the segment slot exactly — enabling frame-accurate audio-video sync without a forced aligner. The implementation supports either cloud TTS or a locally-run 4B-parameter 6-bit quantized model on Apple silicon. The authors publish a qualitative case study on real-time strategy footage alongside a candid list of failure modes: hallucinated game state, resolution loss from mosaicking, and prosody artifacts from time-scaling. Code is released as a reproducible baseline.