FacePlex: 面向对话虚拟人的全双工联合语音-面部运动生成
FacePlex: Toward Natural Full-Duplex Conversational Avatars
AI总结：
提出FacePlex统一流式框架，通过滚动流匹配和滚动交叉注意力实现全双工下语音与面部运动的联合在线生成，在唇同步和运动保真度上优于音频驱动方法。
摘要：
自然的面对面对话需要实时语音生成与同步的面部运动。现有系统仅部分解决此问题：仅语音的全双工模型可以实时生成语音但不产生面部运动，而音频驱动的面部运动模型从已有的音频中动画化面部，而不是在线联合生成语音和运动。为弥补这一差距，我们首先形式化了全双工联合语音-面部运动生成，其中语音令牌和面部运动令牌每一步一起产生。基于此形式，我们提出了FacePlex，一个统一的流式框架，包含两个关键组件。首先，滚动流匹配通过在每个流式步骤提交新的运动帧，将流匹配适应于在线运动生成。其次，滚动交叉注意力将流式音频队列与运动队列耦合，使得语音和面部运动在生成过程中相互条件化。通过大量实验、消融研究和用户研究，我们表明FacePlex能够在在线流式约束下实现全双工联合语音-面部运动生成，同时在唇同步质量和运动保真度上优于音频驱动的面部运动基线。
英文摘要：
Natural human conversation is inherently a real-time interaction in which speech and facial behavior continuously evolve. Enabling such interaction requires a conversational avatar to jointly generate speech and facial motion in real time, prepare facial motion for upcoming speech before the corresponding audio is emitted, and produce non-verbal reactions that reflect the ongoing dialogue. However, existing conversational avatar systems cannot address such requirements: audio-driven methods rely on pre-given speech, while joint streaming generation alone does not ensure anticipatory articulation or semantically appropriate reactions. We propose $\textbf{FacePlex}$, a unified framework for full-duplex speech-facial motion generation and real-time avatar rendering. FacePlex jointly coordinates speech, facial motion, and Gaussian splatting rendering on a shared streaming timeline. To prepare facial motion for upcoming speech, FacePlex predicts a short speech continuation and uses its future hidden states through asymmetric conditioning and denoising, while continuously updating them as new user audio arrives without observing future user input. For dialogue-grounded non-verbal behavior, we construct $\textit{SemReact}$, a dataset aligning dialogue context, reaction semantics, and facial motion, and introduce a semantic behavior router that guides continuous motion during both speaking and listening. Extensive experiments show improved audio-visual synchronization and facial articulation, natural dialogue-grounded non-verbal responses, and low-latency of 122 ms end-to-end avatar interaction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
