简单的词元级置信度提升图像描述正确性
Simple Token-Level Confidence Improves Caption Correctness
AI总结：
本文提出词元级置信度（TLC）方法，通过微调视觉-语言模型并聚合词元置信度来评估图像描述正确性，在多个基准上显著提升准确率并降低物体幻觉。
摘要：
判断一条图像描述是否正确描述图像的能力，是视觉-语言理解中的关键部分。然而，最先进的模型常常误解细粒度细节的正确性，导致输出错误，例如在生成的描述中产生物体幻觉或组合推理能力差。在这项工作中，我们探索了词元级置信度（Token-Level Confidence，简称TLC），作为一种简单却出人意料有效的方法来评估描述的正确性。具体而言，我们在图像描述任务上微调一个视觉-语言模型，将图像和候选描述输入模型，并聚合基于词或序列的代数或学习得到的词元置信度，以估计图像-描述一致性。与预训练模型的序列级分数相比，使用代数置信度度量的TLC在SVO-Probes的动词理解上准确率相对提高了10%，并在Winoground的组合推理中，图像分数和组分数分别相对优于先前最先进水平37%和9%。当训练数据可用时，学习得到的置信度估计器提供了进一步改进的性能，在MS COCO Captions上将物体幻觉率相对原始模型降低了30%，并创造了新的最先进水平。
英文摘要：
The ability to judge whether a caption correctly describes an image is a critical part of vision-language understanding. However, state-of-the-art models often misinterpret the correctness of fine-grained details, leading to errors in outputs such as hallucinating objects in generated captions or poor compositional reasoning. In this work, we explore Token-Level Confidence, or TLC, as a simple yet surprisingly effective method to assess caption correctness. Specifically, we fine-tune a vision-language model on image captioning, input an image and proposed caption to the model, and aggregate either algebraic or learned token confidences over words or sequences to estimate image-caption consistency. Compared to sequence-level scores from pretrained models, TLC with algebraic confidence measures achieves a relative improvement in accuracy by 10% on verb understanding in SVO-Probes and outperforms prior state-of-the-art in image and group scores for compositional reasoning in Winoground by a relative 37% and 9%, respectively. When training data are available, a learned confidence estimator provides further improved performance, reducing object hallucination rates in MS COCO Captions by a relative 30% over the original model and setting a new state-of-the-art.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
