Let Them Talk：音频驱动的多人对话视频生成
Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation
AI总结：
本文提出多人对话视频生成新任务及框架MultiTalk，通过L-RoPE方法解决音频与人物绑定问题，并结合部分参数和多任务训练保留指令遵循能力，在多个数据集上取得优异性能。
摘要：
音频驱动的人体动画方法，如说话头部和说话身体生成，在生成同步的面部动作和具有吸引力的视觉质量视频方面取得了显著进展。然而，现有方法主要关注单个人体动画，难以处理多流音频输入，面临音频与人物之间不正确的绑定问题。此外，它们在指令遵循能力方面表现出局限性。为了解决这个问题，本文提出了一项新任务：多人对话视频生成，并引入了一个新框架 MultiTalk，以应对多人生成过程中的挑战。具体而言，对于音频注入，我们研究了几种方案，并提出了 Label Rotary Position Embedding (L-RoPE) 方法来解决音频与人物的绑定问题。此外，在训练过程中，我们观察到部分参数训练和多任务训练对于保留基础模型的指令遵循能力至关重要。MultiTalk 在多个数据集上（包括说话头部、说话身体和多人数据集）取得了优于其他方法的性能，展示了我们方法强大的生成能力。
英文摘要：
Audio-driven human animation methods, such as talking head and talking body generation, have made remarkable progress in generating synchronized facial movements and appealing visual quality videos. However, existing methods primarily focus on single human animation and struggle with multi-stream audio inputs, facing incorrect binding problems between audio and persons. Additionally, they exhibit limitations in instruction-following capabilities. To solve this problem, in this paper, we propose a novel task: Multi-Person Conversational Video Generation, and introduce a new framework, MultiTalk, to address the challenges during multi-person generation. Specifically, for audio injection, we investigate several schemes and propose the Label Rotary Position Embedding (L-RoPE) method to resolve the audio and person binding problem. Furthermore, during training, we observe that partial parameter training and multi-task training are crucial for preserving the instruction-following ability of the base model. MultiTalk achieves superior performance compared to other methods on several datasets, including talking head, talking body, and multi-person datasets, demonstrating the powerful generation capabilities of our approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
