SemVideo：通过分层语义引导从脑活动重建所观看的内容
SemVideo: Reconstructs What You Watch from Brain Activity via Hierarchical Semantic Guidance
AI总结：
SemVideo通过分层语义引导实现从脑活动到视频的重建，提升语义对齐与时间一致性性能。
摘要：
从脑活动重建动态视觉体验为探索人类视觉感知的神经机制提供了一个有说服力的途径。尽管基于fMRI的图像重建近期取得了显著进展，但将其成功扩展到视频重建仍是一个重大挑战。当前的fMRI到视频重建方法普遍面临两个主要缺点：（i）帧间显著物体的不一致视觉表示导致外观不匹配；（ii）较差的时间一致性，导致运动错位或突兀的帧切换。为了解决这些限制，我们引入了SemVideo，一种由分层语义信息引导的fMRI到视频重建框架。SemVideo的核心是SemMiner，一个分层引导模块，它从原始视频刺激中构建三个层次的语义线索：静态锚点描述、以运动为导向的叙述和整体总结。利用这种语义引导，SemVideo包含三个关键组件：一个语义对齐解码器，它将fMRI信号与从SemMiner导出的CLIP风格嵌入对齐；一个运动适应解码器，它使用新颖的三元注意融合架构来重建动态运动模式；以及一个条件视频渲染器，它利用分层语义引导进行视频重建。在CC2017和HCP数据集上的实验表明，SemVideo在语义对齐和时间一致性方面均实现了优越的性能，为fMRI到视频重建设定了新的状态-of-the-art。
英文摘要：
Reconstructing dynamic visual experiences from brain activity provides a compelling avenue for exploring the neural mechanisms of human visual perception. While recent progress in fMRI-based image reconstruction has been notable, extending this success to video reconstruction remains a significant challenge. Current fMRI-to-video reconstruction approaches consistently encounter two major shortcomings: (i) inconsistent visual representations of salient objects across frames, leading to appearance mismatches; (ii) poor temporal coherence, resulting in motion misalignment or abrupt frame transitions. To address these limitations, we introduce SemVideo, a novel fMRI-to-video reconstruction framework guided by hierarchical semantic information. At the core of SemVideo is SemMiner, a hierarchical guidance module that constructs three levels of semantic cues from the original video stimulus: static anchor descriptions, motion-oriented narratives, and holistic summaries. Leveraging this semantic guidance, SemVideo comprises three key components: a Semantic Alignment Decoder that aligns fMRI signals with CLIP-style embeddings derived from SemMiner, a Motion Adaptation Decoder that reconstructs dynamic motion patterns using a novel tripartite attention fusion architecture, and a Conditional Video Render that leverages hierarchical semantic guidance for video reconstruction. Experiments conducted on the CC2017 and HCP datasets demonstrate that SemVideo achieves superior performance in both semantic alignment and temporal consistency, setting a new state-of-the-art in fMRI-to-video reconstruction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
