arXiv 2603.048032026-03-20cs.CVcs.AIcs.LG

通过对比信号引导扩散重建以实现平衡的视觉表示

Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representation

State Key Laboratory of AI Safety, Institute of Computing Technology, CAS（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；School of Computer Science and Tech., University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Institute of Information Engineering, CAS（信息工程研究所，中国科学院）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）

Boyu Han, Qianqian Xu, Shilong Bao, Zhiyong Yang, Ruochen Cui, Xilin Zhao, Qingming Huang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出通过对比信号优化扩散重建，以提升CLIP中判别能力和细节感知能力，解决传统方法在平衡视觉表示上的不足。

摘要：

视觉编码器在对比学习语言-图像预训练（CLIP）中的有限理解能力成为下游性能的关键瓶颈。这种能力包括判别能力（D-能力）和细节感知能力（P-能力）。最近的解决方案利用扩散模型通过将图像重建条件于CLIP视觉标记来增强表示。我们认为此类范式可能损害D-能力，因此无法有效解决CLIP的表示限制。为了解决这一问题，我们整合对比信号到扩散重建中以追求更全面的视觉表示。我们首先设计了一个简单的方案，通过在输入图像上进行对比学习来增强扩散过程。然而，实验证明，这种简单结合会导致梯度冲突并产生次优性能。为平衡优化，我们引入了扩散对比重建（DCR），统一了学习目标。关键思想是将每个重建图像派生出的对比信号，而非原始输入，注入扩散过程。我们的理论分析表明，DCR损失可以联合优化D-能力与P-能力。在各种基准和多模态大语言模型上的广泛实验验证了我们方法的有效性。代码可在https://github.com/boyuh/DCR获得。

英文摘要：

The limited understanding capacity of the visual encoder in Contrastive Language-Image Pre-training (CLIP) has become a key bottleneck for downstream performance. This capacity includes both Discriminative Ability (D-Ability), which reflects class separability, and Detail Perceptual Ability (P-Ability), which focuses on fine-grained visual cues. Recent solutions use diffusion models to enhance representations by conditioning image reconstruction on CLIP visual tokens. We argue that such paradigms may compromise D-Ability and therefore fail to effectively address CLIP's representation limitations. To address this, we integrate contrastive signals into diffusion-based reconstruction to pursue more comprehensive visual representations. We begin with a straightforward design that augments the diffusion process with contrastive learning on input images. However, empirical results show that the naive combination suffers from gradient conflict and yields suboptimal performance. To balance the optimization, we introduce the Diffusion Contrastive Reconstruction (DCR), which unifies the learning objective. The key idea is to inject contrastive signals derived from each reconstructed image, rather than from the original input, into the diffusion process. Our theoretical analysis shows that the DCR loss can jointly optimize D-Ability and P-Ability. Extensive experiments across various benchmarks and multi-modal large language models validate the effectiveness of our method. The code is available at https://github.com/boyuh/DCR.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。