DiffSpeaker：基于扩散Transformer的语音驱动3D面部动画
DiffSpeaker: Speech-Driven 3D Facial Animation with Diffusion Transformer
AI总结：
DiffSpeaker提出有偏条件注意力模块的Transformer网络，解决语音驱动3D面部动画中数据不足问题，实现高精度唇形同步与快速推理。
摘要：
语音驱动的3D面部动画对许多多媒体应用至关重要。近期工作展示了在该任务中使用扩散模型或Transformer架构的潜力。然而，简单地将两者聚合并未带来性能提升。我们推测这是由于配对音频-4D数据的缺乏，而这类数据对于Transformer在扩散框架内有效充当去噪器至关重要。为解决此问题，我们提出了DiffSpeaker，一个基于Transformer的网络，配备了新颖的有偏条件注意力模块。这些模块替代了标准Transformer中的传统自注意力/交叉注意力，通过精心设计的偏置引导注意力机制聚焦于与任务相关及扩散相关的条件。我们还探讨了扩散范式下精确唇形同步与非言语面部表情之间的权衡。实验表明，我们的模型不仅在现有基准上达到了最先进的性能，而且由于能够并行生成面部运动，推理速度也很快。
英文摘要：
Speech-driven 3D facial animation is important for many multimedia applications. Recent work has shown promise in using either Diffusion models or Transformer architectures for this task. However, their mere aggregation does not lead to improved performance. We suspect this is due to a shortage of paired audio-4D data, which is crucial for the Transformer to effectively perform as a denoiser within the Diffusion framework. To tackle this issue, we present DiffSpeaker, a Transformer-based network equipped with novel biased conditional attention modules. These modules serve as substitutes for the traditional self/cross-attention in standard Transformers, incorporating thoughtfully designed biases that steer the attention mechanisms to concentrate on both the relevant task-specific and diffusion-related conditions. We also explore the trade-off between accurate lip synchronization and non-verbal facial expressions within the Diffusion paradigm. Experiments show our model not only achieves state-of-the-art performance on existing benchmarks, but also fast inference speed owing to its ability to generate facial motions in parallel.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
