arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.22141cs.CL
收藏

通过伪正样本进行相似性标准化以弥合模态差距

Bridging the Modality Gap by Similarity Standardization with Pseudo-Positive Samples

Mitsubishi Electric Corporation(三菱电机公司)

Shuhei Yamashita, Daiki Shirafuji, Tatsuhiko Saito

AI总结:

通过伪正样本进行相似性标准化,有效弥合跨模态检索中的模态差距,提升检索性能。

摘要:

视觉语言模型(VLMs)的进步使得跨模态检索变得有效。然而,当数据库中同时存在文本和图像时,相似度分数会因模态而异。这种现象称为模态差距,会阻碍准确检索。大多数现有研究通过手动标注数据来解决这个问题,例如通过在这些数据上微调VLMs。在本工作中,我们提出了一种通过伪数据构建的相似性标准化方法。我们首先计算每个查询与其配对数据在文本或图像模态之间的相似度分数的均值和方差。利用这些模态特定的统计信息,我们将所有相似度分数标准化,以在不同模态上进行比较。这些统计信息是从伪对中计算得到的,这些伪对是通过检索与每个查询具有最高余弦相似度的文本和图像候选来构建的。我们使用两个多模态问答基准(MMQA和WebQA)在七种VLMs上评估我们的方法,其中每个问题需要检索文本或图像数据。我们的实验结果表明,我们的方法显著提高了检索性能,在查询和目标数据属于不同模态时,MMQA的平均Recall@20提升了64%,WebQA提升了28%。与通过图像描述解决模态差距的E5-V相比,我们确认我们的方法更有效地弥合了模态差距。

英文摘要:

Advances in vision-language models (VLMs) have enabled effective cross-modality retrieval. However, when both text and images exist in the database, similarity scores would differ in scale by modality. This phenomenon, known as the modality gap, hinders accurate retrieval. Most existing studies address this issue with manually labeled data, e.g., by fine-tuning VLMs on them. In this work, we propose a similarity standardization approach with pseudo data construction. We first compute the mean and variance of the similarity scores between each query and its paired data in text or image modality. Using these modality-specific statistics, we standardize all similarity scores to compare on a common scale across modalities. These statistics are calculated from pseudo pairs, which are constructed by retrieving the text and image candidates with the highest cosine similarity to each query. We evaluate our method across seven VLMs using two multi-modal QA benchmarks (MMQA and WebQA), where each question requires retrieving either text or image data. Our experimental results show that our method significantly improves retrieval performance, achieving average Recall@20 gains of 64% on MMQA and 28% on WebQA when the query and the target data belong to different modalities. Compared to E5-V, which addresses the modality gap through image captioning, we confirm that our method more effectively bridges the modality gap.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑