子词清晰度与唇音相关的嵌入式音视频语音增强
Correlating Subword Articulation with Lip Shapes for Embedding Aware Audio-Visual Speech Enhancement
摘要：
在本文中，我们提出了一个视觉嵌入的方法，以提高嵌入感知语音增强（EASE）的同步视觉唇帧在手机和地方的清晰度水平。我们首先提取视觉嵌入唇帧使用预先训练的电话或发音的地方识别器的视觉只EASE（VEASE）。接下来，我们提取视听嵌入噪声语音和唇视频中的信息交叉的方式，利用音频和视觉功能的互补性多模态EASE（MEASE）。在模拟加性噪声污染的TCD-TIMIT语料库上的实验表明，我们提出的基于子词的VEASE方法比传统的词级嵌入方法更有效。此外，在发音位置水平上的视觉嵌入，利用发音位置和嘴唇形状之间的高度相关性，显示出比在电话水平上更好的性能。最后，建议的MEASE框架，结合音频和视频嵌入，产生显着更好的语音质量和清晰度比最好的视觉和音频的EASE系统。
英文摘要：
In this paper, we propose a visual embedding approach to improving embedding aware speech enhancement (EASE) by synchronizing visual lip frames at the phone and place of articulation levels. We first extract visual embedding from lip frames using a pre-trained phone or articulation place recognizer for visual-only EASE (VEASE). Next, we extract audio-visual embedding from noisy speech and lip videos in an information intersection manner, utilizing a complementarity of audio and visual features for multi-modal EASE (MEASE). Experiments on the TCD-TIMIT corpus corrupted by simulated additive noises show that our proposed subword based VEASE approach is more effective than conventional embedding at the word level. Moreover, visual embedding at the articulation place level, leveraging upon a high correlation between place of articulation and lip shapes, shows an even better performance than that at the phone level. Finally the proposed MEASE framework, incorporating both audio and visual embedding, yields significantly better speech quality and intelligibility than those obtained with the best visual-only and audio-only EASE systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
