SAiD：使用扩散模型的语音驱动混合变形面部动画
SAiD: Speech-driven Blendshape Facial Animation with Diffusion
AI总结：
针对语音驱动3D面部动画数据稀缺、唇动单一且编辑繁琐的问题，提出基于Transformer U-Net和扩散模型的SAiD，并构建BlendVOCA数据集，实现更优唇形同步、多样唇动与简化编辑。
摘要：
尽管已有大量研究，但由于缺乏大规模视觉-音频数据集，语音驱动的3D面部动画仍具有挑战性。以往大多数工作通常聚焦于使用最小二乘法在小型数据集上学习回归模型，难以从语音生成多样化的唇部运动，并且需要大量精力来细化生成的输出。为解决这些问题，我们提出了一种采用扩散模型的语音驱动3D面部动画方法SAiD，这是一个轻量级的、基于Transformer的U-Net，具有音频与视觉之间的跨模态对齐偏置，以增强唇形同步。此外，我们推出了BlendVOCA，这是一个由语音音频与混合变形面部模型参数配对组成的基准数据集，以解决公共资源稀缺的问题。我们的实验结果表明，所提方法在唇形同步方面取得了与基线相当或更优的性能，确保了更加多样化的唇部运动，并简化了动画编辑流程。
英文摘要：
Speech-driven 3D facial animation is challenging due to the scarcity of large-scale visual-audio datasets despite extensive research. Most prior works, typically focused on learning regression models on a small dataset using the method of least squares, encounter difficulties generating diverse lip movements from speech and require substantial effort in refining the generated outputs. To address these issues, we propose a speech-driven 3D facial animation with a diffusion model (SAiD), a lightweight Transformer-based U-Net with a cross-modality alignment bias between audio and visual to enhance lip synchronization. Moreover, we introduce BlendVOCA, a benchmark dataset of pairs of speech audio and parameters of a blendshape facial model, to address the scarcity of public resources. Our experimental results demonstrate that the proposed approach achieves comparable or superior performance in lip synchronization to baselines, ensures more diverse lip movements, and streamlines the animation editing process.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
