JoyStreamer: 通过和谐的文本-音频条件化解锁高度表达性的化身
JoyStreamer: Unlocking Highly Expressive Avatars via Harmonized Text-Audio Conditioning
AI总结：
本文提出JoyAvatar框架，通过双教师增强训练算法和动态调节多模态条件，提升视频化身生成的自然性和时序一致性，优于现有模型并在多人群对话等复杂应用中表现突出。
摘要：
现有视频化身模型在说话、演讲和唱歌等场景中表现出色，但大多数方法在文本指令对齐方面有限，尤其在涉及复杂元素如大范围身体运动、动态摄像机轨迹、背景转换或人-物互动时更为明显。为突破这一限制，我们提出了JoyAvatar框架，能够生成长时长的化身视频，具有两个关键技术创新。首先，我们引入了双教师增强训练算法，使模型能够从基础模型中转移固有的文本可控性，同时学习音频-视觉同步。其次，在训练过程中，我们根据不同的去噪时间步动态调节多模态条件（如音频和文本）的强度，以缓解异构条件信号之间的冲突。这两个关键设计显著扩展了化身模型生成自然、时序一致的完整身体运动和动态摄像机运动的能力，同时保持基本的化身能力，如准确的唇形同步和身份一致性。GSB评估结果表明，我们的JoyStreamer模型优于Omnihuman-1.5和KlingAvatar 2.0等现有模型。此外，我们的方法还支持复杂的应用，包括多人对话和非人类主体的角色扮演。一些视频样本可在https://joystreamer.github.io/上提供。
英文摘要：
Existing video avatar models have demonstrated impressive capabilities in scenarios such as talking, public speaking, and singing. However, the majority of these methods exhibit limited alignment with respect to text instructions, particularly when the prompts involve complex elements including large full-body movement, dynamic camera trajectory, background transitions, or human-object interactions. To break out this limitation, we present JoyAvatar, a framework capable of generating long duration avatar videos, featuring two key technical innovations. Firstly, we introduce a twin-teacher enhanced training algorithm that enables the model to transfer inherent text-controllability from the foundation model while simultaneously learning audio-visual synchronization. Secondly, during training, we dynamically modulate the strength of multi-modal conditions (e.g., audio and text) based on the distinct denoising timestep, aiming to mitigate conflicts between the heterogeneous conditioning signals. These two key designs serve to substantially expand the avatar model's capacity to generate natural, temporally coherent full-body motions and dynamic camera movements as well as preserve the basic avatar capabilities, such as accurate lip-sync and identity consistency. GSB evaluation results demonstrate that our JoyStreamer model outperforms the state-of-the-art models such as Omnihuman-1.5 and KlingAvatar 2.0. Moreover, our approach enables complex applications including multi-person dialogues and non-human subjects role-playing. Some video samples are provided on https://joystreamer.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
