通过想象到聆听的MEG映射实现零样本想象语音解码
Zero-Shot Imagined Speech Decoding via Imagined-to-Listened MEG Mapping
AI总结：
本文提出利用聆听语音记录提高想象语音解码的可靠性，通过三阶段流程揭示神经活动与语音刺激的关系，证明想象语音可显著高于随机水平解码。
摘要：
从非侵入性脑记录中解码想象语音具有挑战性，因为想象数据集稀缺且难以跨受试者和会话对齐。本文提出了一种新的解码方法，利用聆听语音时更丰富且可靠标注的记录。我们收集了受过训练的音乐家对节奏旋律和说话刺激的聆听和想象MEG记录。使用受过训练的音乐家有助于改善跨条件的时间对齐。我们随后开发了一个三阶段解码流程，揭示了想象和聆听相同刺激时神经活动之间的一致且有意义的关系。首先，我们训练了六个线性和神经模型，将想象MEG响应映射到聆听响应。我们评估这些模型与未见过的受试者的基线对比，以验证预测的聆听响应保留了刺激特定信息。在第二阶段，我们训练了一个对比词解码器，仅在聆听MEG响应上进行训练，并使用四种嵌入策略进行评估，包括语义、声音和语音表示。在第三阶段，我们通过映射流程处理来自留出受试者的想象MEG响应，计算相应的聆听响应，然后由聆听解码器解码。使用基于排名的分析，我们证明想象的词可以显著高于随机水平解码。我们在此报告了一个概念验证实现的解码结果，所有评估均在留出受试者上进行。我们还展示了随着训练数据量的增加，性能提高，表明这种方法可扩展并可直接应用于现实的脑机接口场景。
英文摘要：
Decoding imagined speech from non-invasive brain recordings is challenging because imagined datasets are scarce and difficult to align temporally across subjects and sessions In this work, we propose a new approach to the decoding of imagined speech that leverages the richer and more reliably labeled recordings during listening to speech. We collected paired listened and imagined MEG recordings to rhythmic melodic and spoken stimuli from trained musicians. Using trained musicians helped improve temporal alignment across conditions. We then developed a three-stage decoding pipeline that revealed consistent and meaningful relationships between neural activity evoked by imagining and listening to the same stimuli. First, we trained six linear and neural models to map imagined MEG responses to listened responses. We evaluated these models against a null baseline from unseen subjects to validate that the predicted-listening responses preserve stimulus-specific information. In the second stage, we trained a contrastive word decoder exclusively on the listened MEG responses, and evaluated it using four embedding strategies including semantic, acoustic, and phonetic representations. In the third stage, we process the imagined MEG responses from held-out subjects through the mapping pipeline to compute the corresponding listening responses that are then decoded by the listened decoder. Using rank-based analysis, we show that the imagined words are decodable significantly above chance. We shall report here the results of a proof-of-concept implementation to decode imagined speech, where all evaluations are performed on held-out subjects. We also demonstrate that performance improves with training data size, suggesting that this approach is scalable and can directly be made applicable to realistic brain-computer interface scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
