arXiv 2512.191152026-05-09cs.CV

生成巨人，检索弱者：为何多模态大语言模型在多模态检索中表现不佳？

Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?

University of Electronic Science and Technology of China（电子科技大学）；Peking University（北京大学）；Tencent Inc（腾讯公司）；Zhongguancun Academy（中关村学院）

Hengyi Feng, Zeang Sheng, Meiyi Qiang, Yang Li, Wentao Zhang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

研究揭示多模态大语言模型在多模态检索中表现不佳的原因，通过稀疏自编码器分析发现其表示空间主要由文本语义构成，视觉语义不足，导致检索性能下降，提出ReAlign方法提升检索效果。

摘要：

尽管多模态大语言模型（MLLMs）在生成任务中表现出色，但在零样本多模态检索任务中却表现出反直觉的不足。本文研究了阻碍MLLMs成为有效检索器的机制。通过稀疏自编码器（SAEs），我们将MLLM输出表示分解为可解释的语义概念以探测其内在行为。分析发现，MLLM的表示空间 overwhelmingly 由文本语义主导；而多模态检索所需的关键视觉语义仅占小部分。我们发现这种不平衡是由于MLLM过度关注图像-文本模态桥接，促进了生成但使嵌入空间同质化，最终降低了多模态检索所需的判别能力。进一步发现，对MLLM相似性计算贡献最大的特定特征组件实际上是干扰项，大大降低了检索性能。基于这些见解，我们提出了ReAlign，一种测试时适应方法，通过白化变换调整MLLM表示空间的几何结构。实验结果表明，这种简单的干预在不同MLLM上一致提升了零样本多模态检索性能，无需微调。代码可在https://github.com/Heinz217/mllm-retrieval-analysis获取。

英文摘要：

Despite the remarkable success of multimodal large language models (MLLMs) in generative tasks, we observe that they exhibit a counterintuitive deficiency in the zero-shot multimodal retrieval task. In this work, we investigate the underlying mechanisms that hinder MLLMs from being effective retrievers. With the help of sparse autoencoders (SAEs), we decompose MLLM output representations into interpretable semantic concepts to probe their intrinsic behavior. Our analysis reveals that the representation space of MLLMs is overwhelmingly dominated by textual semantics; and the visual semantics essential for multimodal retrieval only constitute a small portion. We find that this imbalance is compounded by the heavy focus of MLLMs on bridging image-text modalities, which facilitates generation but homogenizes embeddings and finally diminishes the discriminative power required for multimodal retrieval. We further discover that the specific feature components that contribute most to the similarity computations of MLLMs are actually distractors that greatly reduce retrieval performance. Building on these insights, we propose ReAlign, a test-time adaptation approach that applies a whitening transformation to adjust the geometry of MLLM representation spaces. Empirical results show that this simple intervention consistently improves zero-shot multimodal retrieval performance across diverse MLLMs without fine-tuning efforts. The code is available at https://github.com/Heinz217/mllm-retrieval-analysis.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。