BrainCLIP：通过CLIP桥接大脑与视觉-语言表示，实现通用自然视觉刺激解码
BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding
AI总结：
本文提出任务无关的fMRI脑解码模型BrainCLIP，利用CLIP跨模态能力桥接大脑、图像和文本，在零样本分类、跨模态匹配和图像生成任务上取得显著性能提升，并实现高语义保真重建。
摘要：
由于缺乏配对样本以及功能性磁共振成像（fMRI）信号的低信噪比，从fMRI数据中重建感知的自然图像或解码其语义内容是一项具有挑战性的任务。在这项工作中，我们首次提出了一种任务无关的基于fMRI的脑解码模型BrainCLIP，该模型利用CLIP的跨模态泛化能力来弥合大脑活动、图像和文本之间的模态差距。我们的实验表明，CLIP可以作为通用脑解码任务的枢纽，包括零样本视觉类别解码、fMRI-图像/文本匹配以及fMRI到图像的生成。具体来说，BrainCLIP旨在训练一个映射网络，通过结合视觉和文本监督，将fMRI模式转换为对齐良好的CLIP嵌入空间。我们的实验表明，这种结合可以提升解码模型在fMRI-文本匹配和fMRI到图像生成等特定任务上的性能。在零样本视觉类别解码任务上，BrainCLIP的性能显著优于BraVL，后者是最近提出的专门针对该任务的多模态方法。BrainCLIP还能以高语义保真度重建视觉刺激，并在高层语义特征方面为基于fMRI的自然图像重建树立了新的最先进水平。
英文摘要：
Due to the lack of paired samples and the low signal-to-noise ratio of functional MRI (fMRI) signals, reconstructing perceived natural images or decoding their semantic contents from fMRI data are challenging tasks. In this work, we propose, for the first time, a task-agnostic fMRI-based brain decoding model, BrainCLIP, which leverages CLIP's cross-modal generalization ability to bridge the modality gap between brain activity, image, and text. Our experiments demonstrate that CLIP can act as a pivot for generic brain decoding tasks, including zero-shot visual categories decoding, fMRI-image/text matching, and fMRI-to-image generation. Specifically, BrainCLIP aims to train a mapping network that transforms fMRI patterns into a well-aligned CLIP embedding space by combining visual and textual supervision. Our experiments show that this combination can boost the decoding model's performance on certain tasks like fMRI-text matching and fMRI-to-image generation. On the zero-shot visual category decoding task, BrainCLIP achieves significantly better performance than BraVL, a recently proposed multi-modal method specifically designed for this task. BrainCLIP can also reconstruct visual stimuli with high semantic fidelity and establishes a new state-of-the-art for fMRI-based natural image reconstruction in terms of high-level semantic features.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
