StreamingTalker: 基于音频的3D面部动画与自回归扩散模型
StreamingTalker: Audio-driven 3D Facial Animation with Autoregressive Diffusion Model
AI总结：
本文提出基于自回归扩散模型的StreamingTalker，解决音频驱动3D面部动画中长音频处理延迟和超训练范围的问题，通过动态条件生成高质量实时面部动作。
摘要：
本文聚焦于语音驱动的3D面部动画任务，旨在生成逼真且同步的面部动作，通过音频输入驱动。近期方法采用音频条件扩散模型生成表达自然的动画，但处理整个音频序列存在两个主要挑战：处理超出训练范围的音频序列效果差，且长音频输入会产生显著延迟。为解决这些问题，我们提出一种新的自回归扩散模型，以流式方式处理输入音频。该设计确保了不同音频长度的灵活性，并实现了低延迟，与音频持续时间无关。具体而言，我们选择少量过去帧作为历史动作上下文，并将其与音频输入结合，生成动态条件。该条件引导扩散过程迭代生成面部动作帧，实现高质量的实时合成。此外，我们实现了实时交互演示，突显了该方法的有效性和效率。代码将在https://zju3dv.github.io/StreamingTalker/上发布。
英文摘要：
This paper focuses on the task of speech-driven 3D facial animation, which aims to generate realistic and synchronized facial motions driven by speech inputs. Recent methods have employed audio-conditioned diffusion models for 3D facial animation, achieving impressive results in generating expressive and natural animations. However, these methods process the whole audio sequences in a single pass, which poses two major challenges: they tend to perform poorly when handling audio sequences that exceed the training horizon and will suffer from significant latency when processing long audio inputs. To address these limitations, we propose a novel autoregressive diffusion model that processes input audio in a streaming manner. This design ensures flexibility with varying audio lengths and achieves low latency independent of audio duration. Specifically, we select a limited number of past frames as historical motion context and combine them with the audio input to create a dynamic condition. This condition guides the diffusion process to iteratively generate facial motion frames, enabling real-time synthesis with high-quality results. Additionally, we implemented a real-time interactive demo, highlighting the effectiveness and efficiency of our approach. We will release the code at https://zju3dv.github.io/StreamingTalker/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
