NeuSpeech：将神经信号解码为语音
NeuSpeech: Decode Neural signal as Speech
AI总结：
本文提出首个基于交叉注意力的whisper模型，无需teacher forcing直接从MEG信号生成文本，在GWilliams和Schoffelen数据集上取得60.30和52.89的BLEU-1分数。
摘要：
从大脑动态中解码语言是脑机接口（BCI）领域的一个重要开放方向，特别是考虑到大型语言模型的快速发展。与需要电极植入手术的侵入式信号相比，非侵入式神经信号（如EEG、MEG）因其安全性和通用性而受到越来越多的关注。然而，目前在三个方面的探索尚不充分：1）以往的方法主要关注EEG，但此前没有工作在具有更好信号质量的MEG上解决此问题；2）先前的工作在生成解码期间主要使用“teacher-forcing”，这是不切实际的；3）先前的工作大多基于“BART-based”，并非完全自回归，而完全自回归在其他序列任务中表现更好。在本文中，我们以语音解码的形式探索MEG信号的脑到文本翻译。在此，我们是首个研究基于交叉注意力的“whisper”模型的研究者，该模型无需teacher forcing即可直接从MEG信号生成文本。我们的模型在两个主要数据集（GWilliams和Schoffelen）上，无需预训练和teacher forcing，分别取得了60.30和52.89的令人印象深刻的BLEU-1分数。本文进行了全面回顾，以理解语音解码形式在神经解码任务中的表现，包括预训练初始化、训练与评估集划分、数据增强和缩放规律。代码可在 https://github.com/NeuSpeech/NeuSpeech1 获取。
英文摘要：
Decoding language from brain dynamics is an important open direction in the realm of brain-computer interface (BCI), especially considering the rapid growth of large language models. Compared to invasive-based signals which require electrode implantation surgery, non-invasive neural signals (e.g. EEG, MEG) have attracted increasing attention considering their safety and generality. However, the exploration is not adequate in three aspects: 1) previous methods mainly focus on EEG but none of the previous works address this problem on MEG with better signal quality; 2) prior works have predominantly used $``teacher-forcing"$ during generative decoding, which is impractical; 3) prior works are mostly $``BART-based"$ not fully auto-regressive, which performs better in other sequence tasks. In this paper, we explore the brain-to-text translation of MEG signals in a speech-decoding formation. Here we are the first to investigate a cross-attention-based ``whisper" model for generating text directly from MEG signals without teacher forcing. Our model achieves impressive BLEU-1 scores of 60.30 and 52.89 without pretraining $\&$ teacher-forcing on two major datasets ($\textit{GWilliams}$ and $\textit{Schoffelen}$). This paper conducts a comprehensive review to understand how speech decoding formation performs on the neural decoding tasks, including pretraining initialization, training $\&$ evaluation set splitting, augmentation, and scaling law. Code is available at https://github.com/NeuSpeech/NeuSpeech1$.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
