TaL: a synchronised multi-speaker corpus of ultrasound tongue imaging, audio, and lip videos

Automatic audiovisual synchronisation for ultrasound tongue imaging

Ultrasound tongue imaging is used to visualise the intra-oral articulators during speech production. It is utilised in a range of applications, including speech and language therapy and phonetics research. Ultrasound and speech audio are recorded simultaneously, and in order to correctly use this data, the two modalities should be correctly synchronised. Synchronisation is achieved using specialised hardware at recording time, but this approach can fail in practice resulting in data of limited usability. In this paper, we address the problem of automatically synchronising ultrasound and audio …

基于伪目标生成与域对抗训练的静默舌唇发音语音重建

Speech Reconstruction from Silent Tongue and Lip Articulation By Pseudo Target Generation and Domain Adversarial Training

本文提出基于伪目标生成与域对抗训练的迭代训练方法，从静默舌唇发音中重建语音，相比 TaLNet 基线显著提升可懂度与自然度，WER 降低超15%。

利用跨领域和跨语言的超声舌成像特征进行老年和构音障碍语音识别

Exploiting Cross-domain And Cross-Lingual Ultrasound Tongue Imaging Features For Elderly And Dysarthric Speech Recognition

本文提出跨领域和跨语言的A2A反演方法，利用超声舌成像数据生成发音特征，在老年和构音障碍语音识别中显著降低词或字符错误率，优于仅用声学特征的基线系统。

Synchronising audio and ultrasound by learning cross-modal embeddings

Audiovisual synchronisation is the task of determining the time offset between speech audio and a video recording of the articulators. In child speech therapy, audio and ultrasound videos of the tongue are captured using instruments which rely on hardware to synchronise the two modalities at recording time. Hardware synchronisation can fail in practice, and no mechanism exists to synchronise the signals post hoc. To address this problem, we employ a two-stream neural network which exploits the correlation between the two modalities to find the offset. We train our model on recordings from 69 s…

基于音频条件扩散模型的语音驱动视频编辑

Speech Driven Video Editing via an Audio-Conditioned Diffusion Model

本文提出一种基于音频条件扩散模型的端到端语音驱动视频编辑方法，无需中间结构表示即可同步唇部与下颌运动，并在CREMA-D数据集上验证了可行性。