We estimate articulatory movements in speech production from different modalities - acoustics and phonemes. Acoustic-to articulatory inversion (AAI) is a sequence-to-sequence task. On the other hand, phoneme to articulatory (PTA) motion estimation faces a key challenge in reliably aligning the text and the articulatory movements. To address this challenge, we explore the use of a transformer architecture - FastSpeech, with explicit duration modelling to learn hard alignments between the phonemes and articulatory movements. We also train a transformer model on AAI. We use correlation coefficient (CC) and root mean squared error (rMSE) to assess the estimation performance in comparison to existing methods on both tasks. We observe 154%, 11.8% & 4.8% relative improvement in CC with subject-dependent, pooled and fine-tuning strategies, respectively, for PTA estimation. Additionally, on the AAI task, we obtain 1.5%, 3% and 3.1% relative gain in CC on the same setups compared to the state-of-the-art baseline. We further present the computational benefits of having transformer architecture as representation blocks.
EMA2S: An End-to-End Multimodal Articulatory-to-Speech System
Research Center for Information Technology Innovation, Academia Sinica（中央研究院资讯科技创新研究中心）；Institute of Information Science, Academia Sinica（中央研究院资讯科学研究所）；Graduate School of Informatics, Nagoya University（名古屋大学信息学研究科）；National Institute of Information and Communications Technology（国立信息通信技术研究所）
DNN-based Acoustic-to-Articulatory Inversion using Ultrasound Tongue Imaging
Universidad Industrial de Santander（桑坦德工业大学）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）；MTA-ELTE Lendület Lingual Articulation Research Group（MTA-ELTE Lendület舌部发音研究组）
Speech Synthesis from Text and Ultrasound Tongue Image-based Articulatory Input
Budapest University of Technology and Economics（布达佩斯技术与经济大学）；MTA-ELTE Lendület Lingual Articulation Research Group（MTA-ELTE Lendület 舌部发音研究组）；University of Szeged（塞格德大学）；MTA-SZTE Research Group on Artificial Intelligence（MTA-SZTE 人工智能研究组）；Eötvös Loránd University（厄特沃什·罗兰大学）
Interpretable Modeling of Articulatory Temporal Dynamics from real-time MRI for Phoneme Recognition
Signal Analysis and Interpretation Lab, University of Southern California（南加州大学信号分析与解释实验室）；Department of Linguistics, University of Southern California（南加州大学语言学系）
An Approach to Simultaneous Acquisition of Real-Time MRI Video, EEG, and Surface EMG for Articulatory, Brain, and Muscle Activity During Speech Production
Signal Analysis and Interpretation Laboratory, University of Southern California（南加州大学信号分析与解释实验室）；Ming Hsieh Dept. of Electrical and Computer Engineering, University of Southern California（南加州大学明希斯电气与计算机工程系）；Dept. of Linguistics, University of Southern California（南加州大学语言学系）
Improving Neural Silent Speech Interface Models by Adversarial Training
University of Szeged（塞格德大学）；MTA-SZTE Research Group on Artificial Intelligence（匈牙利科学院-塞格德大学人工智能研究组）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）；Eötvös Loránd University（罗兰大学）
Adaptation of Tongue Ultrasound-Based Silent Speech Interfaces Using Spatial Transformer Networks
University of Szeged（塞格德大学）；ELRN-SZTE Research Group on Artificial Intelligence（ELRN-塞格德大学人工智能研究组）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）
An Audio-textual Diffusion Model For Converting Speech Signals Into Ultrasound Tongue Imaging Data
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Guangdong-Hong Kong-Macao Joint Laboratory of Human-Machine Intelligence-Synergy Systems（粤港澳人机协同智能系统联合实验室）
