概率性语音驱动3D面部运动合成：新基准、方法与应用程序
Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applications
AI总结：
本文针对语音驱动3D面部运动的一对多映射问题，提出大规模概率建模基准与指标，并设计兼具多样性与语音保真度的概率模型，在多个基准上超越现有方法，同时展示了生成未见说话者风格及提升下游视听模型性能的应用价值。
摘要：
我们考虑从语音信号动画化3D面部几何的任务。现有工作主要是确定性的，专注于在具有有限说话者的小型数据集上学习从语音信号到3D面部网格的一对一映射。虽然这些模型可以为训练集中的说话者实现高质量的唇部发音，但它们无法捕捉现实世界中伴随语音的3D面部运动的完整且多样化的分布。重要的是，语音与面部运动之间的关系是一对多的，包含说话者间和说话者内的变化，因此需要一种概率性方法。在本文中，我们识别并解决了迄今为止限制概率模型发展的关键挑战：缺乏适合训练和评估它们的数据集和指标，以及设计一个既能生成多样化结果又能忠实于语音这一强条件信号的模型的困难。我们首先提出了适用于概率建模的大规模基准数据集和指标。然后，我们展示了一个概率模型，该模型同时实现了多样性和对语音的保真度，在所提出的基准上优于其他方法。最后，我们展示了在这些大规模数据集上训练的概率模型的有用应用：我们可以生成多样化的语音驱动3D面部运动，匹配从参考剪辑中提取的未见过的说话者风格；并且我们的合成网格可用于提高下游视听模型的性能。
英文摘要：
We consider the task of animating 3D facial geometry from speech signal. Existing works are primarily deterministic, focusing on learning a one-to-one mapping from speech signal to 3D face meshes on small datasets with limited speakers. While these models can achieve high-quality lip articulation for speakers in the training set, they are unable to capture the full and diverse distribution of 3D facial motions that accompany speech in the real world. Importantly, the relationship between speech and facial motion is one-to-many, containing both inter-speaker and intra-speaker variations and necessitating a probabilistic approach. In this paper, we identify and address key challenges that have so far limited the development of probabilistic models: lack of datasets and metrics that are suitable for training and evaluating them, as well as the difficulty of designing a model that generates diverse results while remaining faithful to a strong conditioning signal as speech. We first propose large-scale benchmark datasets and metrics suitable for probabilistic modeling. Then, we demonstrate a probabilistic model that achieves both diversity and fidelity to speech, outperforming other methods across the proposed benchmarks. Finally, we showcase useful applications of probabilistic models trained on these large-scale datasets: we can generate diverse speech-driven 3D facial motion that matches unseen speaker styles extracted from reference clips; and our synthetic meshes can be used to improve the performance of downstream audio-visual models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
