SemVideo: Reconstructs What You Watch from Brain Activity via Hierarchical Semantic Guidance
| Source: arXiv AI
Tags: brain-computer-interface, fMRI, video-generation, neuroscience, diffusion-models, CLIP
SemVideo reconstructs video clips from fMRI brain activity using a three-tier semantic hierarchy — static anchor descriptions, motion narratives, and holistic summaries — achieving state-of-the-art temporal coherence and semantic alignment on the CC2017 and HCP brain imaging datasets.
Details
Reconstructing what someone watches from their brain activity is a compelling neuroscience problem with implications for brain-computer interfaces and understanding visual perception. The field has made progress on static image reconstruction from fMRI, but video adds a much harder temporal consistency requirement. SemVideo addresses two persistent failures in prior fMRI-to-video work: inconsistent object appearance across frames (identity drift) and poor temporal coherence (abrupt transitions or motion misalignment). The solution is a hierarchical semantic guidance system called SemMiner, which extracts three levels of context from the original video stimulus: static anchor descriptions, motion-oriented narratives, and holistic summaries. Three model components translate this into a reconstruction pipeline: a Semantic Alignment Decoder aligning fMRI signals with CLIP-style embeddings, a Motion Adaptation Decoder with tripartite attention fusion for dynamic patterns, and a Conditional Video Render for final generation. Experiments on CC2017 and HCP datasets show state-of-the-art results in both semantic alignment and temporal consistency. The research is primarily neuroscientific, but the architecture design — using multi-level semantic scaffolding to constrain diffusion generation — could have broader applications in conditional video generation beyond brain imaging.