SynthVSR:利用合成监督扩展视觉语音识别
SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision
AI总结:
本文提出SynthVSR方法,利用语音驱动的唇部动画模型生成合成视觉数据,在仅使用公开数据及有限真实标注的条件下,显著提升了视觉语音识别性能并达到最新SOTA。
摘要:
近期报道的视觉语音识别(VSR)领域最先进的结果通常依赖于日益庞大的视频数据量,而公开可用的转录视频数据集在规模上受到限制。在本文中,我们首次研究了利用合成视觉数据用于VSR的潜力。我们的方法被称为SynthVSR,通过合成的唇部运动大幅提升了VSR系统的性能。SynthVSR背后的关键思想是利用一个语音驱动的唇部动画模型,该模型根据输入语音生成相应的唇部运动。该语音驱动的唇部动画模型在无标签的音视频数据集上进行训练,并且当有标注视频可用时,可以针对预训练的VSR模型进行进一步优化。由于有大量的转录声学数据和面部图像可用,我们能够使用所提出的唇部动画模型生成大规模合成数据,用于半监督VSR训练。我们在最大的公开VSR基准——唇语阅读句子3(Lip Reading Sentences 3, LRS3)上评估了我们的方法性能。SynthVSR仅使用30小时的真实标注数据就达到了43.3%的词错率(WER),优于使用数千小时视频的现成方法。当使用LRS3全部438小时的标注数据时,WER进一步降低至27.9%,与最先进的自监督AV-HuBERT方法不相上下。此外,当与大规模伪标注的音视频数据相结合时,SynthVSR仅使用公开可用数据就取得了16.9%的最新最先进VSR WER,超越了使用多29倍非公开机器转录视频数据(90,000小时)训练的近期最先进方法。最后,我们进行了广泛的消融研究,以了解我们提出的方法中每个组件的作用。
英文摘要:
Recently reported state-of-the-art results in visual speech recognition (VSR) often rely on increasingly large amounts of video data, while the publicly available transcribed video datasets are limited in size. In this paper, for the first time, we study the potential of leveraging synthetic visual data for VSR. Our method, termed SynthVSR, substantially improves the performance of VSR systems with synthetic lip movements. The key idea behind SynthVSR is to leverage a speech-driven lip animation model that generates lip movements conditioned on the input speech. The speech-driven lip animation model is trained on an unlabeled audio-visual dataset and could be further optimized towards a pre-trained VSR model when labeled videos are available. As plenty of transcribed acoustic data and face images are available, we are able to generate large-scale synthetic data using the proposed lip animation model for semi-supervised VSR training. We evaluate the performance of our approach on the largest public VSR benchmark - Lip Reading Sentences 3 (LRS3). SynthVSR achieves a WER of 43.3% with only 30 hours of real labeled data, outperforming off-the-shelf approaches using thousands of hours of video. The WER is further reduced to 27.9% when using all 438 hours of labeled data from LRS3, which is on par with the state-of-the-art self-supervised AV-HuBERT method. Furthermore, when combined with large-scale pseudo-labeled audio-visual data SynthVSR yields a new state-of-the-art VSR WER of 16.9% using publicly available data only, surpassing the recent state-of-the-art approaches trained with 29 times more non-public machine-transcribed video data (90,000 hours). Finally, we perform extensive ablation studies to understand the effect of each component in our proposed method.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。