LEMoN：利用多模态邻居进行标签错误检测
LEMoN: Label Error Detection using Multimodal Neighbors
AI总结：
本文提出LEMoN方法，利用对比预训练多模态模型潜在空间中的多模态邻域自动检测图像-标题数据集中的标签错误，在八个数据集上超过基线3%以上，并显著提升下游标题生成性能。
摘要：
大规模图像-标题配对数据仓库对于视觉-语言模型的发展至关重要。然而，这些数据集通常提取自从网络抓取的噪声数据，包含许多错误标注的实例。为了提高下游模型的可靠性，识别并过滤标题不正确的图像非常重要。然而，除了基于图像-标题嵌入相似度的过滤之外，此前没有工作提出其他方法来过滤噪声多模态数据，或具体评估噪声标题数据对下游训练的影响。在本工作中，我们提出、从理论上论证并实证验证了LEMoN，一种用于识别图像-标题数据集中标签错误的方法。我们的方法利用图像-标题对在对比预训练多模态模型潜在空间中的多模态邻域来自动识别标签错误。通过在八个数据集和十二个基线方法上的实证评估，我们发现LEMoN在标签错误检测上超过基线方法3%以上，并且使用我们的方法过滤后的数据集进行训练，下游标题生成性能比在噪声数据上训练提高了2个BLEU分以上。
英文摘要：
Large repositories of image-caption pairs are essential for the development of vision-language models. However, these datasets are often extracted from noisy data scraped from the web, and contain many mislabeled instances. In order to improve the reliability of downstream models, it is important to identify and filter images with incorrect captions. However, beyond filtering based on image-caption embedding similarity, no prior works have proposed other methods to filter noisy multimodal data, or concretely assessed the impact of noisy captioning data on downstream training. In this work, we propose, theoretically justify, and empirically validate LEMoN, a method to identify label errors in image-caption datasets. Our method leverages the multimodal neighborhood of image-caption pairs in the latent space of contrastively pretrained multimodal models to automatically identify label errors. Through empirical evaluations across eight datasets and twelve baselines, we find that LEMoN outperforms the baselines by over 3% in label error detection, and that training on datasets filtered using our method improves downstream captioning performance by more than 2 BLEU points over noisy training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
