GenHancer：不完美的生成模型实则是强大的以视觉为中心的增强器
GenHancer: Imperfect Generative Models are Secretly Strong Vision-Centric Enhancers
AI总结：
本研究针对CLIP感知细粒度视觉细节不足的问题，从条件机制、去噪配置、生成范式三方面探究生成模型增强表征的关键因素，提出GenHancer方法，在MMVP-VLM基准上取得SOTA，增强后的CLIP可接入多模态大模型提升视觉性能。
摘要：
生成模型与判别模型之间的协同作用正受到越来越多的关注。判别式对比语言-图像预训练（Contrastive Language-Image Pre-Training，CLIP）在高层语义任务中表现出色，但在感知细粒度视觉细节方面存在困难。通常，为了增强表征能力，生成模型会将CLIP的视觉特征作为重建的条件。然而，这一过程的底层原理仍未得到充分探究。在本研究中，我们通过实验发现，视觉上完美的生成结果对于表征增强而言并非总是最优的，其核心在于有效提取生成模型中的细粒度知识，同时抑制无关信息。为探究关键影响因素，我们从三个方面展开深入研究：（1）条件机制：我们发现，即便少量局部标记（token）也会大幅降低重建难度，进而导致训练崩塌。因此我们得出结论，仅使用全局视觉标记作为条件是最有效的策略。（2）去噪配置：我们观察到端到端训练会引入额外信息。为解决这一问题，我们提出两阶段训练策略，以优先学习有用的视觉知识。此外，我们还证明轻量级去噪器也能带来显著的性能提升。（3）生成范式：我们探索了连续去噪器和离散去噪器，均取得了理想结果，验证了我们方法的通用性。通过深入探索，我们最终提出了一种有效方法GenHancer，它在MMVP-VLM基准测试中始终优于现有技术，例如在OpenAICLIP上提升了6.0%。增强后的CLIP还可接入多模态大语言模型，以获得更优的以视觉为中心的性能。所有模型和代码均已公开。
英文摘要：
The synergy between generative and discriminative models receives growing attention. While discriminative Contrastive Language-Image Pre-Training (CLIP) excels in high-level semantics, it struggles with perceiving fine-grained visual details. Generally, to enhance representations, generative models take CLIP's visual features as conditions for reconstruction. However, the underlying principle remains underexplored. In this work, we empirically found that visually perfect generations are not always optimal for representation enhancement. The essence lies in effectively extracting fine-grained knowledge from generative models while mitigating irrelevant information. To explore critical factors, we delve into three aspects: (1) Conditioning mechanisms: We found that even a small number of local tokens can drastically reduce the difficulty of reconstruction, leading to collapsed training. We thus conclude that utilizing only global visual tokens as conditions is the most effective strategy. (2) Denoising configurations: We observed that end-to-end training introduces extraneous information. To address this, we propose a two-stage training strategy to prioritize learning useful visual knowledge. Additionally, we demonstrate that lightweight denoisers can yield remarkable improvements. (3) Generation paradigms: We explore both continuous and discrete denoisers with desirable outcomes, validating the versatility of our method. Through our in-depth explorations, we have finally arrived at an effective method, namely GenHancer, which consistently outperforms prior arts on the MMVP-VLM benchmark, e.g., 6.0% on OpenAICLIP. The enhanced CLIP can be further plugged into multimodal large language models for better vision-centric performance. All the models and codes are made publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
