Animate Your Thoughts：从缓慢脑活动中解耦重建动态自然视觉
Animate Your Thoughts: Decoupled Reconstruction of Dynamic Natural Vision from Slow Brain Activity
AI总结：
针对fMRI时间分辨率低及生成模型幻觉等问题，本文提出两阶段Mind-Animator，通过三模态对比学习与稀疏因果注意力解耦语义、结构和运动特征，并利用膨胀版Stable Diffusion重建视频，取得最先进性能。
摘要：
从脑活动重建人类动态视觉是一项具有重大科学意义的挑战性任务。尽管以往的视频重建方法已经取得实质性进展，但仍存在若干局限，包括：（1）难以同时协调语义（如类别描述）、结构（如尺寸和颜色）以及一致的运动信息（如帧顺序）；（2）fMRI的时间分辨率较低，这给从单个fMRI帧解码视频动态的多个帧带来挑战；（3）对视频生成模型的依赖，使得重建视频中观察到的动态究竟真正来自fMRI数据，还是生成模型产生的幻觉，存在歧义。为克服这些局限，我们提出名为Mind-Animator的两阶段模型。在fMRI到特征阶段，我们从fMRI中解耦语义、结构和运动特征。具体而言，我们采用fMRI-视觉-语言三模态对比学习从fMRI解码语义特征，并设计稀疏因果注意力机制，通过下一帧预测任务解码多帧视频运动特征。在特征到视频阶段，这些特征使用膨胀版Stable Diffusion整合为视频，有效消除外部视频数据干扰。在多个视频-fMRI数据集上的大量实验表明，我们的模型达到最先进性能。全面的可视化分析还从神经生物学角度阐明了模型的可解释性。项目页面：https://mind-animator-design.github.io/。
英文摘要：
Reconstructing human dynamic vision from brain activity is a challenging task with great scientific significance. Although prior video reconstruction methods have made substantial progress, they still suffer from several limitations, including: (1) difficulty in simultaneously reconciling semantic (e.g. categorical descriptions), structure (e.g. size and color), and consistent motion information (e.g. order of frames); (2) low temporal resolution of fMRI, which poses a challenge in decoding multiple frames of video dynamics from a single fMRI frame; (3) reliance on video generation models, which introduces ambiguity regarding whether the dynamics observed in the reconstructed videos are genuinely derived from fMRI data or are hallucinations from generative model. To overcome these limitations, we propose a two-stage model named Mind-Animator. During the fMRI-to-feature stage, we decouple semantic, structure, and motion features from fMRI. Specifically, we employ fMRI-vision-language tri-modal contrastive learning to decode semantic feature from fMRI and design a sparse causal attention mechanism for decoding multi-frame video motion features through a next-frame-prediction task. In the feature-to-video stage, these features are integrated into videos using an inflated Stable Diffusion, effectively eliminating external video data interference. Extensive experiments on multiple video-fMRI datasets demonstrate that our model achieves state-of-the-art performance. Comprehensive visualization analyses further elucidate the interpretability of our model from a neurobiological perspective. Project page: https://mind-animator-design.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
