最近邻归一化提升多模态检索性能
Nearest Neighbor Normalization Improves Multimodal Retrieval
AI总结：
针对多模态预训练模型检索性能仍有不足的问题，提出无需额外训练的最近邻归一化（NNN）方法，可在多个对比模型和数据集上稳定提升图文检索准确率，且无需在参考数据库上训练。
摘要：
多模态模型依托大规模预训练，在图像描述、视觉问答、跨模态检索等任务上取得了出色但仍不完善的性能。本文提出一种无需额外训练即可修正已训练对比式图文检索模型错误的简单高效方法，名为最近邻归一化（Nearest Neighbor Normalization, NNN）。我们在所有测试的对比模型（CLIP、BLIP、ALBEF、SigLIP、BEiT）及所用的两个数据集（MS-COCO和Flickr30k）上，均实现了文本检索和图像检索的指标提升。NNN需要参考数据库，但无需在该数据库上进行任何训练，甚至能提升模型微调后的检索准确率。
英文摘要：
Multimodal models leverage large-scale pre-training to achieve strong but still imperfect performance on tasks such as image captioning, visual question answering, and cross-modal retrieval. In this paper, we present a simple and efficient method for correcting errors in trained contrastive image-text retrieval models with no additional training, called Nearest Neighbor Normalization (NNN). We show an improvement on retrieval metrics in both text retrieval and image retrieval for all of the contrastive models that we tested (CLIP, BLIP, ALBEF, SigLIP, BEiT) and for both of the datasets that we used (MS-COCO and Flickr30k). NNN requires a reference database, but does not require any training on this database, and can even increase the retrieval accuracy of a model after finetuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
