OmniAvatar：具有自适应身体动画的高效音频驱动虚拟形象视频生成
OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animation
AI总结：
OmniAvatar提出像素级多层级音频嵌入与LoRA训练，实现音频驱动的全身视频生成，提升唇形同步和自然运动，并支持文本控制，超越现有模型。
摘要：
音频驱动的人体动画已经取得了显著进展，然而大多数现有方法主要关注面部运动，限制了其生成具有自然同步性和流畅性的全身动画的能力。这些方法在细粒度生成的精确提示控制方面也存在困难。为了应对这些挑战，我们引入了OmniAvatar，一种创新的音频驱动的全身视频生成模型，通过改进唇形同步准确性和自然运动来增强人体动画。OmniAvatar引入了一种像素级多层级音频嵌入策略，以更好地在潜在空间中捕获音频特征，从而增强跨不同场景的唇形同步。为了在有效整合音频特征的同时保留基础模型的提示驱动控制能力，我们采用了基于LoRA的训练方法。大量实验表明，OmniAvatar在面部和半身视频生成方面均超越了现有模型，提供了精确的基于文本的控制，用于创建各种领域的视频，如播客、人际互动、动态场景和歌唱。我们的项目页面是https://omni-avatar.github.io/。
英文摘要：
Significant progress has been made in audio-driven human animation, while most existing methods focus mainly on facial movements, limiting their ability to create full-body animations with natural synchronization and fluidity. They also struggle with precise prompt control for fine-grained generation. To tackle these challenges, we introduce OmniAvatar, an innovative audio-driven full-body video generation model that enhances human animation with improved lip-sync accuracy and natural movements. OmniAvatar introduces a pixel-wise multi-hierarchical audio embedding strategy to better capture audio features in the latent space, enhancing lip-syncing across diverse scenes. To preserve the capability for prompt-driven control of foundation models while effectively incorporating audio features, we employ a LoRA-based training approach. Extensive experiments show that OmniAvatar surpasses existing models in both facial and semi-body video generation, offering precise text-based control for creating videos in various domains, such as podcasts, human interactions, dynamic scenes, and singing. Our project page is https://omni-avatar.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
