MindAlign: 在有限数据下通过多模态嵌入对齐从fMRI信号解码内心语言
MindAlign: Decoding Inner Speech from fMRI Signals via Multimodal Embedding Alignment under Limited Data
AI总结：
提出MindAlign两阶段框架，通过神经-语义对齐将fMRI信号映射到多模态语义空间，再结合视觉上下文冻结多模态语言模型生成文本，在静默图像描述数据上优于基线，且语义-语言投影可跨被试泛化。
摘要：
从非侵入性脑信号解码内心语言仍然是一个基本挑战，因为缺乏显式的语言输出、训练数据有限以及受试者间差异大。现有的脑到文本方法通常依赖于任务特定的解码器微调，这限制了可扩展性并使新参与者的适应复杂化。我们提出了MindAlign，一个解耦的两阶段脑到语言框架，能够在无需修改底层语言模型的情况下，从fMRI信号生成开放式文本。第一阶段学习一个受试者特定的神经-语义对齐，将fMRI活动映射到共享的多模态语义空间，提取内部生成句子的潜在语义草图。第二阶段将该草图与视觉上下文结合，提示冻结的多模态语言模型进行自由形式生成。在静默图像描述期间收集的fMRI数据上的实验表明，所提出的方法始终优于仅fMRI和随机基线。我们进一步表明，学习的语义到语言投影可以跨受试者泛化，当与受试者特定的神经对齐配对时，能够实现有效的解码。这些结果表明，神经信号调节了超出图像驱动先验的语义内容，支持了一种可扩展且模块化的脑到文本解码方向。
英文摘要：
Decoding inner speech from non-invasive brain signals remains a fundamental challenge due to the absence of overt linguistic output, limited training data, and large inter-subject variability. Existing brain-to-text approaches often rely on task-specific decoder fine-tuning, which restricts scalability and complicates adaptation to new participants. We propose MindAlign, a decoupled two-stage brain-to-language framework that enables open-ended text generation from fMRI signals without modifying the underlying language model. The first stage learns a subject-specific neural-semantic alignment that maps fMRI activity into a shared multimodal semantic space, extracting a latent semantic sketch of the internally generated sentence. The second stage integrates this sketch with visual context to prompt a frozen multimodal language model for free-form generation. Experiments on fMRI data collected during silent image description demonstrate that the proposed approach consistently outperforms fMRI-only and random baselines. We further show that the learned semantic-to-language projection can generalize across subjects, enabling effective decoding when paired with subject-specific neural alignment. These results indicate that neural signals modulate semantic content beyond image-driven priors, supporting a scalable and modular direction for brain-to-text decoding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
