立体说话者：基于优先级的混合专家引导的音频驱动3D人类合成
Stereo-Talker: Audio-driven 3D Human Synthesis with Prior-Guided Mixture-of-Experts
AI总结：
Stereo-Talker通过优先级引导的混合专家机制实现音频驱动的3D人类合成，生成具有精确唇同步和逼真质量的视频。
摘要：
本文介绍了一种名为Stereo-Talker的新一代单次音频驱动人类视频合成系统，能够生成具有精确唇同步、表现力强的身体动作、时间一致的逼真质量以及连续视角控制的3D说话视频。该过程遵循两阶段方法。在第一阶段，系统将音频输入映射到高保真运动序列，涵盖上半身动作和面部表情。为了丰富运动多样性和真实性，大语言模型（LLM）先验知识与文本对齐的语义音频特征相结合，利用LLM的跨模态泛化能力来提升运动质量。在第二阶段，我们通过引入优先级引导的混合专家（MoE）机制改进扩散视频生成模型：一个视图引导的MoE专注于视图特定属性，而一个掩码引导的MoE增强了基于区域的渲染稳定性。此外，设计了一个掩码预测模块，从运动数据中推导出人类掩码，提高掩码的稳定性和准确性，并在推理过程中实现掩码引导。我们还介绍了一个包含2203个身份的综合性人类视频数据集，涵盖多样化的身体动作和详细的注释，有助于广泛泛化。代码、数据和预训练模型将发布用于研究目的。
英文摘要：
This paper introduces Stereo-Talker, a novel one-shot audio-driven human video synthesis system that generates 3D talking videos with precise lip synchronization, expressive body gestures, temporally consistent photo-realistic quality, and continuous viewpoint control. The process follows a two-stage approach. In the first stage, the system maps audio input to high-fidelity motion sequences, encompassing upper-body gestures and facial expressions. To enrich motion diversity and authenticity, large language model (LLM) priors are integrated with text-aligned semantic audio features, leveraging LLMs' cross-modal generalization power to enhance motion quality. In the second stage, we improve diffusion-based video generation models by incorporating a prior-guided Mixture-of-Experts (MoE) mechanism: a view-guided MoE focuses on view-specific attributes, while a mask-guided MoE enhances region-based rendering stability. Additionally, a mask prediction module is devised to derive human masks from motion data, enhancing the stability and accuracy of masks and enabling mask guiding during inference. We also introduce a comprehensive human video dataset with 2,203 identities, covering diverse body gestures and detailed annotations, facilitating broad generalization. The code, data, and pre-trained models will be released for research purposes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
