Speech sound disorders are a common communication impairment in childhood. Because speech disorders can negatively affect the lives and the development of children, clinical intervention is often recommended. To help with diagnosis and treatment, clinicians use instrumented methods such as spectrograms or ultrasound tongue imaging to analyse speech articulations. Analysis with these methods can be laborious for clinicians, therefore there is growing interest in its automation. In this paper, we investigate the contribution of ultrasound tongue imaging for the automatic detection of speech articulation errors. Our systems are trained on typically developing child speech and augmented with a database of adult speech using audio and ultrasound. Evaluation on typically developing speech indicates that pre-training on adult speech and jointly using ultrasound and audio gives the best results with an accuracy of 86.9%. To evaluate on disordered speech, we collect pronunciation scores from experienced speech and language therapists, focusing on cases of velar fronting and gliding of /r/. The scores show good inter-annotator agreement for velar fronting, but not for gliding errors. For automatic velar fronting error detection, the best results are obtained when jointly using ultrasound and audio. The best system correctly detects 86.6% of the errors identified by experienced clinicians. Out of all the segments identified as errors by the best system, 73.2% match errors identified by clinicians. Results on automatic gliding detection are harder to interpret due to poor inter-annotator agreement, but appear promising. Overall findings suggest that automatic detection of speech articulation errors has potential to be integrated into ultrasound intervention software for automatically quantifying progress during speech therapy.
An Audio-textual Diffusion Model For Converting Speech Signals Into Ultrasound Tongue Imaging Data
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Guangdong-Hong Kong-Macao Joint Laboratory of Human-Machine Intelligence-Synergy Systems（粤港澳人机协同智能系统联合实验室）
Incorporating Ultrasound Tongue Images for Audio-Visual Speech Enhancement
National Engineering Research Center of Speech and Language Information Processing（国家语言信息处理工程研究中心）；University of Science and Technology of China（中国科学技术大学）
Incorporating Ultrasound Tongue Images for Audio-Visual Speech Enhancement through Knowledge Distillation
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China（中国科学技术大学国家语言语音信息处理工程研究中心）
TaL: a synchronised multi-speaker corpus of ultrasound tongue imaging, audio, and lip videos
University of Edinburgh（爱丁堡大学）；University of Science and Technology of China（中国科学技术大学）；Queen Margaret University（玛格丽特女王大学）；Articulate Instruments Ltd.（Articulate仪器有限公司）
我们提出了舌头和嘴唇语料库（TaL），音频，超声舌头成像和嘴唇视频的多扬声器语料库。TaL由两部分组成：TaL 1是一个专业的语音人才，英语为母语的男性的一套六个录音会话; TaL 80是一个没有语音人才经验的81个英语为母语的录音会话。总体而言，语料库包含24小时的并行超声、视频和音频数据，其中大约13.5小时是语音。本文介绍了语料库，并提出了基准测试结果的任务，语音识别，语音合成（发音声学映射），和自动同步的超声音频。TaL语料库在CC BY-NC 4.0许可下公开可用。
Quantification of Transducer Misalignment in Ultrasound Tongue Imaging
Budapest University of Technology and Economics（布达佩斯技术与经济大学）；MTA-ELTE Lendület Lingual Articulation Research Group（匈牙利科学院-罗兰大学伦杜莱特语言发音研究组）；National University of Defense Technology（国防科技大学）
Speech Synthesis from Text and Ultrasound Tongue Image-based Articulatory Input
Budapest University of Technology and Economics（布达佩斯技术与经济大学）；MTA-ELTE Lendület Lingual Articulation Research Group（MTA-ELTE Lendület 舌部发音研究组）；University of Szeged（塞格德大学）；MTA-SZTE Research Group on Artificial Intelligence（MTA-SZTE 人工智能研究组）；Eötvös Loránd University（厄特沃什·罗兰大学）
Adaptation of Tacotron2-based Text-To-Speech for Articulatory-to-Acoustic Mapping using Ultrasound Tongue Imaging
Budapest University of Technology and Economics（布达佩斯科技经济大学）；University of Szeged（塞格德大学）；MTA-SZTE Research Group on Artificial Intelligence（匈牙利科学院-塞格德大学人工智能研究组）；Eötvös Loránd University（厄特沃什·罗兰大学）；MTA-ELTE Lendület Lingual Articulation Research Group（匈牙利科学院-厄特沃什·罗兰大学林杜莱特舌部发音研究组）
Adaptation of Tongue Ultrasound-Based Silent Speech Interfaces Using Spatial Transformer Networks
University of Szeged（塞格德大学）；ELRN-SZTE Research Group on Artificial Intelligence（ELRN-塞格德大学人工智能研究组）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）
DNN-based Acoustic-to-Articulatory Inversion using Ultrasound Tongue Imaging
Universidad Industrial de Santander（桑坦德工业大学）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）；MTA-ELTE Lendület Lingual Articulation Research Group（MTA-ELTE Lendület舌部发音研究组）
Ultrasound-based Silent Speech Interface Built on a Continuous Vocoder
Budapest University of Technology and Economics（布达佩斯技术与经济大学）；MTA-SZTE Research Group on Artificial Intelligence（匈牙利科学院-塞格德大学人工智能研究组）；University of Szeged（塞格德大学）
Ultrasound-based Articulatory-to-Acoustic Mapping with WaveGlow Speech Synthesis
Budapest University of Technology and Economics（布达佩斯技术与经济大学）；MTA-ELTE Lendület Lingual Articulation Research Group（匈牙利科学院-厄特沃什·罗兰大学伦杜莱特语言发音研究组）；University of Szeged（塞格德大学）；MTA-SZTE Research Group on Artificial Intelligence（匈牙利科学院-塞格德大学人工智能研究组）；Eötvös Loránd University（厄特沃什·罗兰大学）
