AnyTalker: 通过交互细化扩展多人物谈话视频生成
AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement
AI总结：
AnyTalker通过引入身份感知注意力机制和高效训练流程，实现了多人物谈话视频生成的高可扩展性和自然交互性。
摘要：
最近，多人物视频生成开始获得关注。尽管已有初步工作探索了音频驱动的多人物谈话视频生成，但它们常常面临挑战，因为收集多样化的多人物数据成本高，且驱动多个身份的连贯交互困难。为解决这些问题，我们提出了AnyTalker，一个多人物生成框架，其特征是可扩展的多流处理架构。具体来说，我们扩展了扩散变换器的注意力块，引入了新的身份感知注意力机制，该机制迭代处理身份-音频对，允许任意扩展可驱动的身份。此外，训练多人物生成模型需要大量的多人物数据。我们提出的训练流程仅依赖单人视频来学习多人物说话模式，并通过少量真实的多人物片段来细化交互性。此外，我们贡献了一个专门设计的指标和数据集，用于评估生成多人物视频的自然性和交互性。广泛的实验表明，AnyTalker在唇部同步、视觉质量和自然交互性方面取得了显著成果，实现了数据成本与身份可扩展性之间的良好平衡。
英文摘要：
Recently, multi-person video generation has started to gain prominence. While a few preliminary works have explored audio-driven multi-person talking video generation, they often face challenges due to the high costs of diverse multi-person data collection and the difficulty of driving multiple identities with coherent interactivity. To address these challenges, we propose AnyTalker, a multi-person generation framework that features an extensible multi-stream processing architecture. Specifically, we extend Diffusion Transformer's attention block with a novel identity-aware attention mechanism that iteratively processes identity-audio pairs, allowing arbitrary scaling of drivable identities. Besides, training multi-person generative models demands massive multi-person data. Our proposed training pipeline depends solely on single-person videos to learn multi-person speaking patterns and refines interactivity with only a few real multi-person clips. Furthermore, we contribute a targeted metric and dataset designed to evaluate the naturalness and interactivity of the generated multi-person videos. Extensive experiments demonstrate that AnyTalker achieves remarkable lip synchronization, visual quality, and natural interactivity, striking a favorable balance between data costs and identity scalability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
