通过并行音素序列预测增强基于 EEG 的所听语音解码
Enhancing Listened Speech Decoding from EEG via Parallel Phoneme Sequence Prediction
AI总结：
本文提出在 EEG 所听语音解码中加入并行辅助音素预测器，使模型可同时输出语音波形与文本音素序列，并在两种模态上超越以往方法。
摘要：
脑机接口（BCI）提供了许多以人为本的应用可能性，尤其会影响神经系统疾病患者。从大脑活动中解码文本或语音是一个相关领域，能够提升言语感知受损人群的生活质量。我们提出一种新方法，通过利用一个辅助音素预测器同时解码文本音素序列，来增强从脑电图（EEG）信号中解码所听语音的能力。所提出的模型架构由三个主要部分组成：EEG 模块、语音模块和音素预测器。EEG 模块学习将 EEG 信号恰当地表示为 EEG 嵌入。语音模块根据这些 EEG 嵌入生成语音波形。音素预测器以文本模态输出解码后的音素序列。我们提出的方法使用户能够同时从 EEG 信号获得两种模态（语音波形和文本音素序列）的解码所听语音，无需为每种模态使用串联的顺序流水线。该方法在两种模态上也优于以往方法。源代码和语音样本均已公开发布。
英文摘要：
Brain-computer interfaces (BCI) offer numerous human-centered application possibilities, particularly affecting people with neurological disorders. Text or speech decoding from brain activities is a relevant domain that could augment the quality of life for people with impaired speech perception. We propose a novel approach to enhance listened speech decoding from electroencephalography (EEG) signals by utilizing an auxiliary phoneme predictor that simultaneously decodes textual phoneme sequences. The proposed model architecture consists of three main parts: EEG module, speech module, and phoneme predictor. The EEG module learns to properly represent EEG signals into EEG embeddings. The speech module generates speech waveforms from the EEG embeddings. The phoneme predictor outputs the decoded phoneme sequences in text modality. Our proposed approach allows users to obtain decoded listened speech from EEG signals in both modalities (speech waveforms and textual phoneme sequences) simultaneously, eliminating the need for a concatenated sequential pipeline for each modality. The proposed approach also outperforms previous methods in both modalities. The source code and speech samples are publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
