Silent speech interfaces (SSI) aim to reconstruct the speech signal from a recording of the articulatory movement, such as an ultrasound video of the tongue. Currently, deep neural networks are the most successful technology for this task. The efficient solution requires methods that do not simply process single images, but are able to extract the tongue movement information from a sequence of video frames. One option for this is to apply recurrent neural structures such as the long short-term memory network (LSTM) in combination with 2D convolutional neural networks (CNNs). Here, we experiment with another approach that extends the CNN to perform 3D convolution, where the extra dimension corresponds to time. In particular, we apply the spatial and temporal convolutions in a decomposed form, which proved very successful recently in video action recognition. We find experimentally that our 3D network outperforms the CNN+LSTM model, indicating that 3D CNNs may be a feasible alternative to CNN+LSTM networks in SSI systems.
Ultrasound-based Silent Speech Interface Built on a Continuous Vocoder
Budapest University of Technology and Economics（布达佩斯技术与经济大学）；MTA-SZTE Research Group on Artificial Intelligence（匈牙利科学院-塞格德大学人工智能研究组）；University of Szeged（塞格德大学）
Autoencoder-Based Articulatory-to-Acoustic Mapping for Ultrasound Silent Speech Interfaces
MTA-SZTE Research Group on Artificial Intelligence（MTA-SZTE人工智能研究组）；University of Szeged（塞格德大学）；Eötvös Loránd University（厄特沃什·罗兰大学）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）
Neural Speaker Embeddings for Ultrasound-based Silent Speech Interfaces
University of Szeged（塞格德大学）；MTA-SZTE Research Group on Artificial Intelligence（匈牙利科学院-塞格德大学人工智能研究组）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）
Adaptation of Tongue Ultrasound-Based Silent Speech Interfaces Using Spatial Transformer Networks
University of Szeged（塞格德大学）；ELRN-SZTE Research Group on Artificial Intelligence（ELRN-塞格德大学人工智能研究组）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）
Cross-Modal Masking for Robust Silent Speech Synthesis Using sEMG and Lipreading
Aholab research group within the HiTZ Center at University of the Basque Country (UPV/EHU)（巴斯克大学HiTZ中心内Aholab研究组）；PRHLT research center, Universitat Politècnica de València (UPV)（瓦伦西亚理工大学PRHLT研究中心）
Harnessing AI for Speech Reconstruction using Multi-view Silent Video Feed
Adobe Systems（奥多比系统公司）；Netaji Subhas Institute of Technology（内塔吉·苏巴斯理工学院）；National Institute of Informatics（国立信息学研究所）；National University of Singapore（新加坡国立大学）
Speech Synthesis from Text and Ultrasound Tongue Image-based Articulatory Input
Budapest University of Technology and Economics（布达佩斯技术与经济大学）；MTA-ELTE Lendület Lingual Articulation Research Group（MTA-ELTE Lendület 舌部发音研究组）；University of Szeged（塞格德大学）；MTA-SZTE Research Group on Artificial Intelligence（MTA-SZTE 人工智能研究组）；Eötvös Loránd University（厄特沃什·罗兰大学）
Incorporating Ultrasound Tongue Images for Audio-Visual Speech Enhancement through Knowledge Distillation
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China（中国科学技术大学国家语言语音信息处理工程研究中心）
IR-UWB Radar-Based Contactless Silent Speech Recognition of Vowels, Consonants, Words, and Phrases
School of Integrated Technology, Yonsei University（延世大学综合技术学院）；NVIDIA Corporation（英伟达公司）；Department of Convergence IT Engineering, Pohang University of Science and Technology（浦项科技大学融合IT工程系）
Incorporating Ultrasound Tongue Images for Audio-Visual Speech Enhancement
National Engineering Research Center of Speech and Language Information Processing（国家语言信息处理工程研究中心）；University of Science and Technology of China（中国科学技术大学）
Affect Decoding in Phonated and Silent Speech Production from Surface EMG
1 CHI -- Chair of Health Informatics, TUM University Hospital, Munich, Germany 2 SAIL -- Signal Analysis；Interpretation Lab, University of Southern California, USA 3 CSL -- Cognitive Systems Lab, University of Bremen, Germany 4 GLAM -- Group on Language, Audio, \& Music, Imperial College London, UK 5 MCML -- Munich Center for Machine Learning, Germany
Speaking Without Sound: Multi-speaker Silent Speech Voicing with Facial Inputs Only
Music and Audio Research Group (MARG)（音乐与音频研究组）；Department of Intelligence and Information（智能与信息系）；Interdisciplinary Program in Artificial Intelligence（人工智能跨学科项目）；Artificial Intelligence Institute, Seoul National University（首尔国立大学人工智能研究所）
Ultrasound-based Articulatory-to-Acoustic Mapping with WaveGlow Speech Synthesis
Budapest University of Technology and Economics（布达佩斯技术与经济大学）；MTA-ELTE Lendület Lingual Articulation Research Group（匈牙利科学院-厄特沃什·罗兰大学伦杜莱特语言发音研究组）；University of Szeged（塞格德大学）；MTA-SZTE Research Group on Artificial Intelligence（匈牙利科学院-塞格德大学人工智能研究组）；Eötvös Loránd University（厄特沃什·罗兰大学）
An Audio-textual Diffusion Model For Converting Speech Signals Into Ultrasound Tongue Imaging Data
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Guangdong-Hong Kong-Macao Joint Laboratory of Human-Machine Intelligence-Synergy Systems（粤港澳人机协同智能系统联合实验室）
Improving Neural Silent Speech Interface Models by Adversarial Training
University of Szeged（塞格德大学）；MTA-SZTE Research Group on Artificial Intelligence（匈牙利科学院-塞格德大学人工智能研究组）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）；Eötvös Loránd University（罗兰大学）
Budapest University of Technology and Economics（布达佩斯技术与经济大学）；University of Szeged, Institute of Informatics（塞格德大学信息学院）；HUN-REN–SZTE Research Group on Artificial Intelligence（HUN-REN–SZTE人工智能研究组）
Adaptation of Tacotron2-based Text-To-Speech for Articulatory-to-Acoustic Mapping using Ultrasound Tongue Imaging
Budapest University of Technology and Economics（布达佩斯科技经济大学）；University of Szeged（塞格德大学）；MTA-SZTE Research Group on Artificial Intelligence（匈牙利科学院-塞格德大学人工智能研究组）；Eötvös Loránd University（厄特沃什·罗兰大学）；MTA-ELTE Lendület Lingual Articulation Research Group（匈牙利科学院-厄特沃什·罗兰大学林杜莱特舌部发音研究组）
TaL: a synchronised multi-speaker corpus of ultrasound tongue imaging, audio, and lip videos
University of Edinburgh（爱丁堡大学）；University of Science and Technology of China（中国科学技术大学）；Queen Margaret University（玛格丽特女王大学）；Articulate Instruments Ltd.（Articulate仪器有限公司）
我们提出了舌头和嘴唇语料库（TaL），音频，超声舌头成像和嘴唇视频的多扬声器语料库。TaL由两部分组成：TaL 1是一个专业的语音人才，英语为母语的男性的一套六个录音会话; TaL 80是一个没有语音人才经验的81个英语为母语的录音会话。总体而言，语料库包含24小时的并行超声、视频和音频数据，其中大约13.5小时是语音。本文介绍了语料库，并提出了基准测试结果的任务，语音识别，语音合成（发音声学映射），和自动同步的超声音频。TaL语料库在CC BY-NC 4.0许可下公开可用。
Quantification of Transducer Misalignment in Ultrasound Tongue Imaging
Budapest University of Technology and Economics（布达佩斯技术与经济大学）；MTA-ELTE Lendület Lingual Articulation Research Group（匈牙利科学院-罗兰大学伦杜莱特语言发音研究组）；National University of Defense Technology（国防科技大学）
Automated Measurement of Geniohyoid Muscle Thickness During Speech Using Deep Learning and Ultrasound
Biomedical Engineering Department, Hong Kong Polytechnic University, Hong Kong, China（生物医学工程系，香港理工大学，香港，中国）；English and Communication Department, Hong Kong Polytechnic University, Hong Kong, China（英语与交流系，香港理工大学，香港，中国）；Department of Language Science and Technology, Hong Kong Polytechnic University, Hong Kong, China（语言科学与技术系，香港理工大学，香港，中国）；Research Institute for Smart Ageing, Hong Kong Polytechnic University, Hong Kong, China（智能老龄化研究学院，香港理工大学，香港，中国）
