Department of XXX, University of YYY, Location, Country(XXX系,YYY大学,地点,国家);School of ZZZ, Institute of WWW, Location, Country(ZZZ学院,WWW研究所,地点,国家);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Department of Software Convergence, Kyung Hee University(软件融合系,庆熙大学)
Yiqian Yang, Hyejeong Jo, Yiqun Duan, Qiang Zhang, Jinni Zhou, Xuming Hu, Won Hee Lee, Renjing Xu, Hui Xiong
Deciphering language from brain activity is a crucial task in brain-computer interface (BCI) research. Non-invasive cerebral signaling techniques including electroencephalography (EEG) and magnetoencephalography (MEG) are becoming increasingly popular due to their safety and practicality, avoiding invasive electrode implantation. However, current works under-investigated three points: 1) a predominant focus on EEG with limited exploration of MEG, which provides superior signal quality; 2) poor performance on unseen text, indicating the need for models that can better generalize to diverse linguistic contexts; 3) insufficient integration of information from other modalities, which could potentially constrain our capacity to comprehensively understand the intricate dynamics of brain activity.
This study presents a novel approach for translating MEG signals into text using a speech-decoding framework with multiple alignments. Our method is the first to introduce an end-to-end multi-alignment framework for totally unseen text generation directly from MEG signals. We achieve an impressive BLEU-1 score on the \textit{GWilliams} dataset, significantly outperforming the baseline from 5.49 to 6.86 on the BLEU-1 metric. This improvement demonstrates the advancement of our model towards real-world applications and underscores its potential in advancing BCI research. Code is available at $\href{https://github.com/NeuSpeech/MAD-MEG2text}{https://github.com/NeuSpeech/MAD-MEG2text}$.
Transfer Learning from ImageNet for MEG-Based Decoding of Imagined Speech
Université Paris Cité(巴黎西岱大学);Institut Pasteur(巴斯德研究所);AP-HP(巴黎公立医院集团);INSERM(法国国家健康与医学研究院);CNRS(法国国家科学研究中心);Fondation Pour l’Audition(听力基金会);Institut de l’Audition(听力研究所);IHU reConnect(reConnect医院大学研究所);University of Geneva(日内瓦大学)
Decoding Phone Pairs from MEG Signals Across Speech Modalities
HiTZ Center, University of the Basque Country -- UPV/EHU(巴斯克国家大学希茨中心);Basque Centre on Cognition, Brain and Language -- BCBL(巴斯克认知、大脑与语言中心);Laboratory of Functional Anatomy, Faculty of Human Motor Sciences, Université libre de Bruxelles (ULB)(人类运动科学系功能解剖实验室);Laboratoire de Neuroanatomie et Neuroimagerie translationnelles (LN2T), ULB Neuroscience Institute, Université libre de Bruxelles (ULB)(神经解剖与神经影像翻译实验室(LN2T)、ULB神经科学研究所);WEL Research Institute(WEL研究机构)
Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing
State Key Laboratory of Multimodal Artificial Intelligence System, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Department of Computer Science, The University of Manchester(曼彻斯特大学计算机科学系);Department of Language Science and Technology, Hong Kong Polytechnic University(香港理工大学语言科学与技术系)
Brain-to-Text Decoding: A Non-invasive Approach via Typing
Meta AI;École Normale Supérieure, Université PSL, CNRS(巴黎高等师范学院、巴黎文理研究大学、法国国家科学研究中心);Hospital Foundation Adolphe de Rothschild(阿道夫·德·罗斯柴尔德医院基金会);Basque Center on Cognition, Brain and Language(巴斯克认知、大脑与语言中心);Ikerbasque, Basque Foundation for Science(伊克尔巴斯克基金会(巴斯克科学基金会))
Accurate Decoding of Natural Sentences from Non-Invasive Brain Recordings
Meta AI;École Normale Supérieure(高等师范学院);Université PSL(巴黎文理大学);CNRS(法国国家科学研究中心);Hospital Foundation Adolphe de Rothschild(阿道夫·德·罗斯柴尔德医院基金会);Univ. Lille(里尔大学);Basque Center on Cognition, Brain and Language(巴斯克认知、大脑与语言中心);Paris Cité University(巴黎城市大学);Inria(法国国家信息与自动化研究所);Université Paris-Saclay(巴黎萨克雷大学);INSERM(法国国家健康与医学研究院);CEA(法国原子能和替代能源委员会)