DreamID-Omni: 一体化框架用于可控的人-centric音频视频生成
DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation
AI总结：
DreamID-Omni提出了一种统一框架，通过双层解耦策略和多任务训练方案，实现对可控人-centric音频视频生成的高效控制。
摘要：
近年来，基础模型的进步彻底改变了联合音频视频生成。然而，现有方法通常将包括基于参考的音频视频生成（R2AV）、视频编辑（RV2AV）和音频驱动视频动画（RA2V）等以人为中心的任务视为孤立目标。此外，在单一框架内实现对多个角色身份和声音特质的精确、解耦控制仍是一个开放性挑战。在本文中，我们提出了DreamID-Omni，一种用于可控的人-centric音频视频生成的一体化框架。具体而言，我们设计了一种对称条件扩散变换器，通过对称条件注入方案整合异构条件信号。为解决多人物场景中普遍存在的身份-声音特质绑定失败和说话者混淆问题，我们引入了双层解耦策略：在信号层面采用同步RoPE以确保刚性注意力空间绑定，在语义层面采用结构化描述以建立显式属性-主体映射。此外，我们设计了一种多任务渐进训练方案，利用弱约束生成先验来正则化强约束任务，防止过拟合并协调不同目标。大量实验表明，DreamID-Omni在视频、音频和音频-视觉一致性方面均实现了最先进的性能，甚至超过了领先的商业模型。我们将发布代码以弥合学术研究与商业级应用之间的差距。
英文摘要：
Recent advancements in foundation models have revolutionized joint audio-video generation. However, existing approaches typically treat human-centric tasks including reference-based audio-video generation (R2AV), video editing (RV2AV) and audio-driven video animation (RA2V) as isolated objectives. Furthermore, achieving precise, disentangled control over multiple character identities and voice timbres within a single framework remains an open challenge. In this paper, we propose DreamID-Omni, a unified framework for controllable human-centric audio-video generation. Specifically, we design a Symmetric Conditional Diffusion Transformer that integrates heterogeneous conditioning signals via a symmetric conditional injection scheme. To resolve the pervasive identity-timbre binding failures and speaker confusion in multi-person scenarios, we introduce a Dual-Level Disentanglement strategy: Synchronized RoPE at the signal level to ensure rigid attention-space binding, and Structured Captions at the semantic level to establish explicit attribute-subject mappings. Furthermore, we devise a Multi-Task Progressive Training scheme that leverages weakly-constrained generative priors to regularize strongly-constrained tasks, preventing overfitting and harmonizing disparate objectives. Extensive experiments demonstrate that DreamID-Omni achieves comprehensive state-of-the-art performance across video, audio, and audio-visual consistency, even outperforming leading proprietary commercial models. We will release our code to bridge the gap between academic research and commercial-grade applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
