LVSum: A Benchmark for Timestamp-Aware Long Video Summarization
| Source: Apple ML Research
Tags: LVSum, Apple ML Research, video summarization, MLLM, temporal grounding, multimodal, benchmark, computer vision
Apple ML Research's LVSum benchmark — 72 videos across 13 domains averaging 16 minutes — reveals that transcripts outperform visual frames for MLLM summarization and that all tested models fail systematically at temporal grounding, exposing a critical gap in long video AI evaluation.
Details
Apple researchers Alkesh Patel, Melis Ozyildirim, Ying-Chang Cheng, and Ganesh Nagarajan published LVSum, a human-annotated benchmark targeting a specific weakness in multimodal large language models: long-form video summarization with accurate temporal alignment. The dataset covers 72 videos across 13 diverse domains, with an average duration of 16 minutes and up to 10 human-authored summaries per video, each containing explicit temporal references. The team evaluated leading proprietary and open-source MLLMs using two newly introduced LLM-based metrics — content relevance and modality coherence — alongside standard automatic metrics. Three findings stand out: transcripts contribute substantially more to summarization quality than visual frames alone, suggesting visual understanding in long videos is weaker than commonly assumed; a significant performance gap persists between model outputs and human-written summaries; and current MLLMs exhibit systematic weaknesses in temporal grounding, instruction adherence, and cross-modal coherence. These results matter beyond benchmarking — they clarify where video AI systems actually fail in production contexts. Enterprises deploying video understanding pipelines for meeting summarization, surveillance, media indexing, or training data curation face these exact gaps. Apple released source code on GitHub alongside the paper. The benchmark joins a growing body of evaluation infrastructure exposing the limits of current video MLLMs, particularly their tendency to rely on text transcripts rather than genuine visual comprehension of temporal sequences.