arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.14715cs.CVcs.CL
收藏

FINEMATCH:基于方面的细粒度图文不匹配检测与纠正

FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correction

University of Rochester(罗切斯特大学);Adobe Research(奥多比研究院)

Hang Hua, Jing Shi, Kushal Kafle, Simon Jenni, Daoan Zhang, John Collomosse, Scott Cohen, Jiebo Luo

AI总结:

提出FineMatch基准,用于基于方面的细粒度图文不匹配检测与纠正,并引入ITM-IoU评估指标,实验表明其可提升VLM的细粒度匹配能力并支持幻觉检测。

摘要:

大规模预训练的最新进展推动了先进的视觉语言模型(VLM)的发展,这些模型在理解和生成多模态内容方面表现出卓越的能力。尽管VLM具有执行复杂推理的出色能力,但当前模型往往难以有效且精确地捕捉图像和文本两侧的组合信息。为了解决这一问题,我们提出了FineMatch,一个新的基于方面的细粒度文本与图像匹配基准,专注于文本和图像的不匹配检测与纠正。该基准引入了一项新任务,用于提升和评估VLM在基于方面的细粒度文本与图像匹配中的组合能力。在此任务中,模型需要识别标题中不匹配的方面短语,确定方面的类别,并为可能包含0到3个不匹配的图像-文本对提出纠正建议。为了评估模型在此新任务上的性能,我们提出了一种新的评估指标,名为ITM-IoU,我们的实验表明该指标与人工评估高度相关。此外,我们还对现有主流VLM进行了全面的实验分析,包括全监督学习和上下文学习设置。我们发现,在FineMatch上训练的模型在检测细粒度文本与图像不匹配方面表现出更强的熟练度。此外,具有强大多模态上下文学习能力的模型(例如GPT-4V、Gemini Pro Vision)在细粒度组合图像与文本匹配分析方面并不那么熟练。借助FineMatch,我们能够构建一个用于文本到图像生成幻觉检测与纠正的系统。

英文摘要:

Recent progress in large-scale pre-training has led to the development of advanced vision-language models (VLMs) with remarkable proficiency in comprehending and generating multimodal content. Despite the impressive ability to perform complex reasoning for VLMs, current models often struggle to effectively and precisely capture the compositional information on both the image and text sides. To address this, we propose FineMatch, a new aspect-based fine-grained text and image matching benchmark, focusing on text and image mismatch detection and correction. This benchmark introduces a novel task for boosting and evaluating the VLMs' compositionality for aspect-based fine-grained text and image matching. In this task, models are required to identify mismatched aspect phrases within a caption, determine the aspect's class, and propose corrections for an image-text pair that may contain between 0 and 3 mismatches. To evaluate the models' performance on this new task, we propose a new evaluation metric named ITM-IoU for which our experiments show a high correlation to human evaluation. In addition, we also provide a comprehensive experimental analysis of existing mainstream VLMs, including fully supervised learning and in-context learning settings. We have found that models trained on FineMatch demonstrate enhanced proficiency in detecting fine-grained text and image mismatches. Moreover, models (e.g., GPT-4V, Gemini Pro Vision) with strong abilities to perform multimodal in-context learning are not as skilled at fine-grained compositional image and text matching analysis. With FineMatch, we are able to build a system for text-to-image generation hallucination detection and correction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑