CREM: 为多模态检索与理解的压缩驱动表示增强
CREM: Compression-Driven Representation Enhancement for Multimodal Retrieval and Comprehension
AI总结：
CREM通过压缩驱动的表示增强方法，在提升多模态检索性能的同时保持生成能力，实现生成与嵌入任务的统一优化。
摘要：
多模态大语言模型（MLLMs）在视觉描述和视觉问答等理解任务中表现出显著的成功。然而，它们直接应用于基于嵌入的任务如检索仍然具有挑战性，因为输出格式和优化目标之间存在差异。先前的方法通常采用对比微调来适应MLLMs进行检索，但以牺牲其生成能力为代价。我们主张，生成和嵌入任务本质上依赖于共享的认知机制，特别是跨模态表示对齐和上下文理解。为此，我们提出了CREM（压缩驱动表示增强模型），其统一框架在增强多模态表示以进行检索的同时保持生成能力。具体而言，我们引入了一种基于压缩的提示设计，利用可学习的合唱令牌来聚合多模态语义，并引入了一种压缩驱动的训练策略，通过压缩感知注意力整合对比和生成目标。广泛的实验表明，CREM在MMEB上实现了最先进的检索性能，同时在多个理解基准上保持了强大的生成性能。我们的发现表明，生成监督可以在所提出的压缩驱动范式下进一步提高MLLMs的表示质量。
英文摘要：
Multimodal Large Language Models (MLLMs) have shown remarkable success in comprehension tasks such as visual description and visual question answering. However, their direct application to embedding-based tasks like retrieval remains challenging due to the discrepancy between output formats and optimization objectives. Previous approaches often employ contrastive fine-tuning to adapt MLLMs for retrieval, but at the cost of losing their generative capabilities. We argue that both generative and embedding tasks fundamentally rely on shared cognitive mechanisms, specifically cross-modal representation alignment and contextual comprehension. To this end, we propose CREM (Compression-driven Representation Enhanced Model), with a unified framework that enhances multimodal representations for retrieval while preserving generative ability. Specifically, we introduce a compression-based prompt design with learnable chorus tokens to aggregate multimodal semantics and a compression-driven training strategy that integrates contrastive and generative objectives through compression-aware attention. Extensive experiments demonstrate that CREM achieves state-of-the-art retrieval performance on MMEB while maintaining strong generative performance on multiple comprehension benchmarks. Our findings highlight that generative supervision can further improve the representational quality of MLLMs under the proposed compression-driven paradigm.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
