SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation
| Source: arXiv AI
Tags: SemComp-Bench, video generation, benchmarking, multimodal, computer vision
SemComp-Bench introduces outcome-oriented evaluation for video generation — measuring whether models complete intended tasks with semantic fidelity, not just visual realism. Experiments show current models consistently fail to achieve both task goals and semantic grounding simultaneously.
Details
Video generation benchmarks have long rewarded visual quality and appearance consistency, but fail to measure whether a model actually accomplishes the task it was asked to do. SemComp-Bench reframes evaluation around Semantic Task Completion: success requires achieving the intended outcome while maintaining semantic correspondence to a reference image.\n\nThe benchmark introduces SemComp-Data, a dataset spanning six domains, built through a four-stage curation pipeline that converts raw videos into standardized instances. Each instance includes a reference image, detailed and brief instructions, and an outcome-centric clip. A VLM-based evaluation protocol scores models on OA (Outcome Achievement) and GR (Generation Reliability) using structured binary questions.\n\nExperiments on representative video generation models confirm the task is hard: models trained to produce realistic video struggle to combine semantic fidelity to the reference with correct task outcomes. This reveals a meaningful gap between how models are optimized and what instruction-following video generation actually requires.\n\nFor practitioners building video generation pipelines — particularly for agentic or automated applications — SemComp-Bench offers a more practically relevant measure than appearance-based metrics. The dataset covers six domains, making it broadly applicable across robotics, cooking, and other task-oriented scenarios.