以往的视听语音分离方法利用视频中说话人面部动作与语音的同步性，以自监督方式对语音分离进行监督。本文提出一种同时借助人脸和手语解决语音分离问题的模型，我们将该问题称为扩展语音分离问题。我们设计了一个通用深度学习网络，用于学习音频、人脸、手语信息三种模态的组合，以更好地解决语音分离问题。为训练该模型，我们引入了一个名为Chinese Sign Language News Speech（CSLNSpeech，中国手语新闻语音）数据集的大规模数据集，其中同时存在音频、人脸、手语三种模态。实验结果表明，所提模型比常规视听系统具备更优的性能与鲁棒性。此外，手语模态也可单独用于监督语音分离任务，引入手语对听障人士的学习与交流有帮助。最后，我们的模型是一个通用语音分离框架，在两个开源视听数据集上能取得极具竞争力的分离性能。代码可在https://github.com/iveveive/SLNSpeech 获取。
英文摘要：
Previous audio-visual speech separation methods use the synchronization of the speaker's facial movement and speech in the video to supervise the speech separation in a self-supervised way. In this paper, we propose a model to solve the speech separation problem assisted by both face and sign language, which we call the extended speech separation problem. We design a general deep learning network for learning the combination of three modalities, audio, face, and sign language information, for better solving the speech separation problem. To train the model, we introduce a large-scale dataset named the Chinese Sign Language News Speech (CSLNSpeech) dataset, in which three modalities of audio, face, and sign language coexist. Experiment results show that the proposed model has better performance and robustness than the usual audio-visual system. Besides, sign language modality can also be used alone to supervise speech separation tasks, and the introduction of sign language is helpful for hearing-impaired people to learn and communicate. Last, our model is a general speech separation framework and can achieve very competitive separation performance on two open-source audio-visual datasets. The code is available at https://github.com/iveveive/SLNSpeech
AISHELL6-whisper: A Chinese Mandarin Audio-visual Whisper Speech Dataset with Speech Recognition Baselines
School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；Beijing AISHELL Technology Co., Ltd.（北京AISHELL科技有限公司）；AI Center, OPPO（OPPO人工智能中心）
Magnetoencephalography (MEG) Based Non-Invasive Chinese Speech Decoding
Ministry of Education Key Laboratory of Image Processing and Intelligent Control（教育部图像处理与智能控制重点实验室）；School of Artificial Intelligence and Automation（人工智能与自动化学院）；Huazhong University of Science and Technology（华中科技大学）；Department of Neurosurgery（神经外科科）；Tongji Hospital（同济医院）；Tongji Medical College（同济医学院）
Institute of Psychology（心理研究所）；University of the Chinese Academy of Sciences（中国科学院大学）；Beijing Union University（北京联合大学）；Jiangsu University of Science and Technology（江苏科技大学）；University of Science and Technology of China（中国科学技术大学）
Gipsa-lab, University Grenoble Alpes（格勒诺布尔阿尔卑斯大学Gipsa-lab）；Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）深圳大数据研究院）；Grenoble Institute of Technology（格勒诺布尔理工学院）
Perceiver-Prompt: Flexible Speaker Adaptation in Whisper for Chinese Disordered Speech Recognition
Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；The Chinese University of Hong Kong（香港中文大学）；Shenzhen Institute of Advanced Technology, CAS（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）
A Chinese Heart Failure Status Speech Database with Universal and Personalised Classification
School of Information Science and Technology（信息科学与技术学院）；Advanced Computing and Storage Laboratory, 2012 Laboratories（先进计算与存储实验室）；Institute of High Performance Computing（高性能计算研究所）
Connected Speech-Based Cognitive Assessment in Chinese and English
Usher Institute, Medical School（医学院厄舍研究所）；School of Health in Social Science（社会科学健康学院）；School of Engineering, University of Edinburgh（爱丁堡大学工程学院）；Carnegie Mellon University（卡内基梅隆大学）；University of Delaware（特拉华大学）；Cardinal Tien Hospital（圣保禄医院）
Cross-Modal Mutual Learning for Cued Speech Recognition
Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）深圳大数据研究院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
An End-to-end Chinese Text Normalization Model based on Rule-guided Flat-Lattice Transformer
Tsinghua-CUHK Joint Research Center for Media Sciences, Technologies and Systems（清华-中大媒体科学技术与系统联合研究中心）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；The Chinese University of Hong Kong（香港中文大学）；Databaker (Beijing) Technology Co., Ltd（标贝（北京）科技有限公司）
BASPRO: a balanced script producer for speech corpus collection based on the genetic algorithm
Research Center for Information Technology Innovation, Academia Sinica（中央研究院资讯科技创新研究中心）；Columbia University（哥伦比亚大学）；Institute of Information Science, Academia Sinica（中央研究院资讯科学研究所）
Fine-grained Speech Sentiment Analysis in Chinese Psychological Support Hotlines Based on Large-scale Pre-trained Model
School of Software Engineering, Beijing University of Technology（北京工业大学软件学院）；Peking University Huilongguan Clinical Medical School（北京大学回龙湖临床医学院）；Sorbonne Université（索邦大学）；Institut du Cerveau – Paris Brain Institute - ICM（巴黎脑研究所）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Inserm（法国国家健康与医学研究院）；AP-HP（巴黎公共医疗援助医院）；Hôpital de la Pitié-Salpêtrière（皮蒂-萨尔佩特里埃医院）
DiM-Gestor: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2
School of Media Engineering, Communication University of Zhejiang（浙江传媒学院媒体工程学院）；School of Broadcast Announcing Arts, Communication University of Zhejiang（浙江传媒学院播音主持艺术学院）；Faculty of Humanities and Arts, Macau University of Science and Technology（澳门科技大学人文艺术学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Zhejiang Lab（之江实验室）；Changjiang Academy of Art and Design, Shantou University（汕头大学长江艺术与设计学院）
