Advances in vision-language models (VLMs) have enabled effective cross-modality retrieval. However, when both text and images exist in the database, similarity scores would differ in scale by modality. This phenomenon, known as the modality gap, hinders accurate retrieval. Most existing studies address this issue with manually labeled data, e.g., by fine-tuning VLMs on them. In this work, we propose a similarity standardization approach with pseudo data construction. We first compute the mean and variance of the similarity scores between each query and its paired data in text or image modality. Using these modality-specific statistics, we standardize all similarity scores to compare on a common scale across modalities. These statistics are calculated from pseudo pairs, which are constructed by retrieving the text and image candidates with the highest cosine similarity to each query. We evaluate our method across seven VLMs using two multi-modal QA benchmarks (MMQA and WebQA), where each question requires retrieving either text or image data. Our experimental results show that our method significantly improves retrieval performance, achieving average Recall@20 gains of 64% on MMQA and 28% on WebQA when the query and the target data belong to different modalities. Compared to E5-V, which addresses the modality gap through image captioning, we confirm that our method more effectively bridges the modality gap.
Bridging the Modality Gap for Speech-to-Text Translation
National Laboratory of Pattern Recognition, Institute of Automation, CAS(中国科学院自动化研究所模式识别国家重点实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);CAS Center for Excellence in Brain Science and Intelligence Technology(中国科学院脑科学与智能技术卓越创新中心)
Interpreting the linear structure of vision-language model embedding spaces
Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University(哈佛大学自然与人工智能研究所);Department of Computer Science, Harvard University(哈佛大学计算机科学系)
ChartReasoner: Code-Driven Modality Bridging for Long-Chain Reasoning in Chart Question Answering
MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所MAIS研究所);Shenyang Institute of Computing Technology, Chinese Academy of Sciences(中国科学院沈阳计算技术研究所);Beijing Wenge Technology Co., Ltd.(北京文耕科技有限公司);University of Chinese Academy of Sciences(中国科学院大学)
MCEN: Bridging Cross-Modal Gap between Cooking Recipes and Dish Images with Latent Variable Model
Zhejiang University(浙江大学);Singapore Management University(新加坡管理大学);Alibaba-Zhejiang University Joint Institute of Frontier Technologies(阿里巴巴-浙江大学前沿技术联合研究院)
Interpretable probabilistic embeddings: bridging the gap between topic models and neural networks
National Research University Higher School of Economics(国立研究大学高等经济学院);Lomonosov Moscow State University(莫斯科国立大学);Moscow Institute of Physics and Technology(莫斯科物理技术学院)
CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale
Simon Fraser University(西蒙弗雷泽大学);Aalborg University(奥胡斯大学);Vector Institute(向量研究所);University of Guelph(圭尔夫大学);Alberta Machine Intelligence Institute (Amii)(阿尔伯塔人工智能研究所)
Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillation
Columbia University(哥伦比亚大学);University of Washington(华盛顿大学)
Bridging Text and Molecule: A Survey on Multimodal Frameworks for Molecule
Center for Research on Intelligent Perception and Computing (CRIPAC)(智能感知与计算研究中心);State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS)(多模态人工智能系统国家重点实验室);Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)