arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.01512cs.CL
收藏

MAD: 多对齐的MEG到文本解码

MAD: Multi-Alignment MEG-to-Text Decoding

Department of XXX, University of YYY, Location, Country(XXX系,YYY大学,地点,国家);School of ZZZ, Institute of WWW, Location, Country(ZZZ学院,WWW研究所,地点,国家);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Department of Software Convergence, Kyung Hee University(软件融合系,庆熙大学)

Yiqian Yang, Hyejeong Jo, Yiqun Duan, Qiang Zhang, Jinni Zhou, Xuming Hu, Won Hee Lee, Renjing Xu, Hui Xiong

AI总结:

MAD通过多对齐框架实现从MEG信号到未知文本的端到端解码,显著提升BLEU-1分数,推动BCI研究发展。

摘要:

从脑活动解码语言是脑机接口(BCI)研究中的关键任务。非侵入性脑信号技术,包括脑电图(EEG)和磁脑图(MEG),由于其安全性和实用性,越来越受欢迎,避免了侵入性电极植入。然而,现有工作在三个方面研究不足:1)主要关注EEG,对MEG的探索有限,而MEG提供更优的信号质量;2)在未见过的文本上表现差,表明需要能够更好地泛化到多样语言环境的模型;3)对其他模态信息的整合不足,这可能限制我们全面理解脑活动复杂动态的能力。本研究提出了一种使用语音解码框架和多对齐方法将MEG信号转换为文本的新方法。我们的方法是首个引入端到端多对齐框架,直接从MEG信号生成完全未知文本的模型。我们在\textit{GWilliams}数据集上取得了令人印象深刻BLEU-1分数,显著优于基线模型,BLEU-1指标从5.49提升到6.86。这一进步表明了模型在实际应用中的进步,并突显了其在BCI研究中的潜力。代码可在$\href{https://github.com/NeuSpeech/MAD-MEG2text}{https://github.com/NeuSpeech/MAD-MEG2text}$上获得。

英文摘要:

Deciphering language from brain activity is a crucial task in brain-computer interface (BCI) research. Non-invasive cerebral signaling techniques including electroencephalography (EEG) and magnetoencephalography (MEG) are becoming increasingly popular due to their safety and practicality, avoiding invasive electrode implantation. However, current works under-investigated three points: 1) a predominant focus on EEG with limited exploration of MEG, which provides superior signal quality; 2) poor performance on unseen text, indicating the need for models that can better generalize to diverse linguistic contexts; 3) insufficient integration of information from other modalities, which could potentially constrain our capacity to comprehensively understand the intricate dynamics of brain activity. This study presents a novel approach for translating MEG signals into text using a speech-decoding framework with multiple alignments. Our method is the first to introduce an end-to-end multi-alignment framework for totally unseen text generation directly from MEG signals. We achieve an impressive BLEU-1 score on the \textit{GWilliams} dataset, significantly outperforming the baseline from 5.49 to 6.86 on the BLEU-1 metric. This improvement demonstrates the advancement of our model towards real-world applications and underscores its potential in advancing BCI research. Code is available at $\href{https://github.com/NeuSpeech/MAD-MEG2text}{https://github.com/NeuSpeech/MAD-MEG2text}$.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑