Bind-Your-Avatar：基于动态3D掩码嵌入路由的多说话角色视频生成
Bind-Your-Avatar: Multi-Talking-Character Video Generation with Dynamic 3D-mask-based Embedding Router
AI总结：
提出Bind-Your-Avatar，一种基于MM-DiT的多角色同场景说话视频生成模型，通过3D掩码嵌入路由器绑定音频与角色，并构建首个专用数据集与基准，实验验证其优越性能。
摘要：
近年来，音频驱动的说话头生成技术取得了显著进展。然而，现有方法主要聚焦于单角色场景。尽管一些方法能够生成两个个体之间的独立对话视频，但生成多个物理上共处同一空间环境、共享同一空间场景的角色进行统一对话视频这一关键挑战仍未得到充分解决。该设置面临两个关键挑战：音频与角色之间的对应控制，以及缺乏包含同一场景内多角色说话视频的合适数据集。为应对这些挑战，我们提出了Bind-Your-Avatar，一种基于MM-DiT的模型，专门用于同一场景下的多说话角色视频生成。具体而言，我们提出：（1）一种新颖框架，包含细粒度嵌入路由器，将“谁”和“说什么”绑定在一起，以解决音频与角色对应控制问题。（2）两种实现3D掩码嵌入路由器的方法，该路由器能够实现逐帧、细粒度的个体角色控制，并基于观测到的几何先验采用不同的损失函数，以及一种掩码细化策略，以增强预测掩码的准确性和时间平滑性。（3）据我们所知，首个专门为多说话角色视频生成构建的数据集，并附带开源数据处理流水线，以及（4）一个用于双说话角色视频生成的基准，通过大量实验证明其性能优于多种最先进方法。
英文摘要：
Recent years have witnessed remarkable advances in audio-driven talking head generation. However, existing approaches predominantly focus on single-character scenarios. While some methods can create separate conversation videos between two individuals, the critical challenge of generating unified conversation videos with multiple physically co-present characters sharing the same spatial environment remains largely unaddressed. This setting presents two key challenges: audio-to-character correspondence control and the lack of suitable datasets featuring multi-character talking videos within the same scene. To address these challenges, we introduce Bind-Your-Avatar, an MM-DiT-based model specifically designed for multi-talking-character video generation in the same scene. Specifically, we propose (1) A novel framework incorporating a fine-grained Embedding Router that binds `who' and `speak what' together to address the audio-to-character correspondence control. (2) Two methods for implementing a 3D-mask embedding router that enables frame-wise, fine-grained control of individual characters, with distinct loss functions based on observed geometric priors and a mask refinement strategy to enhance the accuracy and temporal smoothness of the predicted masks. (3) The first dataset, to the best of our knowledge, specifically constructed for multi-talking-character video generation, and accompanied by an open-source data processing pipeline, and (4) A benchmark for the dual-talking-characters video generation, with extensive experiments demonstrating superior performance over multiple state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
