通过有向场景图对大型视觉-语言模型进行全面图像描述生成的基准测试
Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning
AI总结：
本文提出CompreCap基准，利用有向场景图从对象覆盖、属性准确性和关系得分等多层面评估大型视觉-语言模型的详细图像描述生成能力，实验验证其与人工评估高度一致。
摘要：
生成理解图像中富含文本的视觉内容的详细描述，已引起大型视觉-语言模型（LVLMs）越来越多的关注。然而，很少有研究开发专门针对详细描述的基准来衡量其准确性和全面性。在本文中，我们引入了一个名为CompreCap的详细描述基准，从有向场景图的角度评估视觉上下文。具体而言，我们首先根据常见对象词汇表，将图像手动分割成语义上有意义的区域（即语义分割掩码），同时区分所有这些区域内对象的属性。然后，标注这些对象的定向关系标签，以构成一个有向场景图，该图能够很好地编码图像的丰富组合信息。基于我们的有向场景图，我们开发了一个流程，从多个层面评估LVLMs生成的详细描述，包括对象级覆盖率、属性描述的准确性、关键关系的得分等。在CompreCap数据集上的实验结果证实，我们的评估方法与跨LVLMs的人工评估得分高度一致。
英文摘要：
Generating detailed captions comprehending text-rich visual content in images has received growing attention for Large Vision-Language Models (LVLMs). However, few studies have developed benchmarks specifically tailored for detailed captions to measure their accuracy and comprehensiveness. In this paper, we introduce a detailed caption benchmark, termed as CompreCap, to evaluate the visual context from a directed scene graph view. Concretely, we first manually segment the image into semantically meaningful regions (i.e., semantic segmentation mask) according to common-object vocabulary, while also distinguishing attributes of objects within all those regions. Then directional relation labels of these objects are annotated to compose a directed scene graph that can well encode rich compositional information of the image. Based on our directed scene graph, we develop a pipeline to assess the generated detailed captions from LVLMs on multiple levels, including the object-level coverage, the accuracy of attribute descriptions, the score of key relationships, etc. Experimental results on the CompreCap dataset confirm that our evaluation method aligns closely with human evaluation scores across LVLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
