arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.25328cs.CVcs.MM
收藏

DIVA: 利用统一多模态模型中的表示差异实现相互增强

DIVA: Harnessing the Representation Divergence in Unified Multimodal Models for Mutual Reinforcement

Ping An Technology (Shenzhen) Co., Ltd., Shenzhen, China(平安科技(深圳)有限公司,深圳,中国);University of Science and Technology of China(中国科学技术大学)

Renjie Lu, Xulong Zhang, Xiaoyang Qu, Shangfei Wang, Jianzong Wang

AI总结:

针对统一多模态模型中理解与生成任务因监督信号差异导致相互干扰的问题,提出DIVA框架,通过分解视觉表示为共享和独有成分并利用互信息估计实现内部协同,在理解与生成任务上分别提升7.82%和8.46%。

摘要:

基于单一架构构建的统一多模态模型(UMMs)在理解和生成任务中均展现出令人印象深刻的表现。我们识别出一个基本挑战,即由不同监督信号引起的归纳偏差:生成分支偏好能够重建的高保真、细粒度表示,而理解分支则偏好对任务无关因素保持不变的语义判别性嵌入。因此,在单一骨干网络中优化这些互补但不等价的目标会导致相互损害而非增强。在本文中,我们首先分析了统一骨干网络中这种干扰的根本原因,并揭示了其内部表示中的互补结构。受此观察启发,我们提出了DIVA,一个自我改进的训练后框架,将表示差异转化为内部协同。通过基于两条互补信息流将视觉表示显式分解为共享和独有成分,DIVA使得理解和生成分支都能实现有益的迁移,同时通过互信息估计保护独有信息免受跨流干扰的完整性。尽管具有通用性,我们的方法在视觉理解(+7.82%)和生成(+8.46%)任务上均取得了一致的改进。官方代码见:https://github.com/Jayyy-H/DIVA。

英文摘要:

Unified Multimodal models (UMMs) built on a single architecture have shown impressive performance in both understanding and generation. We identify a fundamental challenge that lies in inductive biases induced by distinct supervision signals: generation branch prefers high-fidelity, fine-grained representations capable of reconstruction, while the understanding favours semantically discriminative embeddings that remain invariant to task-irrelevant factors. Consequently, optimizing these complementary but non-equivalent objectives within a monolithic backbone leads to mutual impairment instead of enhancement. In this paper, we first analyze the root cause of this interference in unified backbones and reveal a complementary structure in their internal representations. Motivated by the observation, we propose DIVA, a self-improved post-training framework that transforms the representation divergence into interior synergy. By explicitly factorizing the visual representation into shared and unique components based on two complementary information flow, DIVA enables both the understanding and generation branches to achieve beneficial transferring while preserving the integrity of unique information from cross-flow interference via mutual information estimation. Despite its generality, our method consistently achieves improvements across visual understanding (+7.82%) and generation (+8.46%). The official code is available at: https://github.com/Jayyy-H/DIVA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑