通过排序和基于LLM的融合提升图像字幕描述的丰富性
Improving Image Captioning Descriptiveness by Ranking and LLM-based Fusion
AI总结：
本文提出一种无需额外训练的图像字幕增强方法，通过BLIPScore排序并利用LLM融合多个SoTA模型生成的字幕，在MS-COCO和Flickr30k上提升了描述丰富性并减少幻觉。
摘要：
最先进的（SoTA）图像字幕模型通常在微软上下文中的常见对象（MS-COCO）数据集上进行训练，该数据集包含平均长度约为十个词元的人工标注字幕。尽管这些短字幕对于一般场景理解有效，但往往无法捕捉复杂场景并传达详细信息。此外，字幕模型倾向于对“平均”字幕产生偏差，即只捕捉更一般的方面，从而忽略更精细的细节。在本文中，我们提出了一种新颖的方法，通过结合来自不同SoTA字幕模型生成的字幕，来生成更丰富、更具信息量的图像字幕。我们提出的方法无需额外的模型训练：给定一张图像，它利用文献中的预训练模型生成初始字幕，然后使用我们新引入的基于图像-文本的度量标准（我们称之为BLIPScore）对它们进行排序。随后，使用大型语言模型（LLM）融合排名前两位的字幕，以生成最终更详细的描述。在MS-COCO和Flickr30k测试集上的实验结果表明，根据ALOHa、CAPTURE和Polos度量，我们的方法在字幕-图像对齐和幻觉减少方面具有有效性。一项主观研究进一步支持了这些结果，表明我们的模型生成的字幕通常被认为更符合人类判断。通过结合不同SoTA模型的优势，我们的方法提高了图像字幕的质量和吸引力，弥合了自动化系统与人类生成描述的丰富性和信息性之间的差距。这一进展使得为视觉-语言模型和字幕模型的训练生成更合适的字幕成为可能。
英文摘要：
State-of-The-Art (SoTA) image captioning models are often trained on the MicroSoft Common Objects in Context (MS-COCO) dataset, which contains human-annotated captions with an average length of approximately ten tokens. Although effective for general scene understanding, these short captions often fail to capture complex scenes and convey detailed information. Moreover, captioning models tend to exhibit bias towards the ``average'' caption, which captures only the more general aspects, thus overlooking finer details. In this paper, we present a novel approach to generate richer and more informative image captions by combining the captions generated from different SoTA captioning models. Our proposed method requires no additional model training: given an image, it leverages pre-trained models from the literature to generate the initial captions, and then ranks them using a newly introduced image-text-based metric, which we name BLIPScore. Subsequently, the top two captions are fused using a Large Language Model (LLM) to produce the final, more detailed description. Experimental results on the MS-COCO and Flickr30k test sets demonstrate the effectiveness of our approach in terms of caption-image alignment and hallucination reduction according to the ALOHa, CAPTURE, and Polos metrics. A subjective study lends additional support to these results, suggesting that the captions produced by our model are generally perceived as more consistent with human judgment. By combining the strengths of diverse SoTA models, our method enhances the quality and appeal of image captions, bridging the gap between automated systems and the rich and informative nature of human-generated descriptions. This advance enables the generation of more suitable captions for the training of both vision-language and captioning models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
