IRR：用于评估视觉语言模型的图像评论排序框架
IRR: Image Review Ranking Framework for Evaluating Vision-Language Models
AI总结：
提出IRR框架，通过图像评论排序评估视觉语言模型与人类判断的一致性，发现现有模型相关性不足，需改进评估方法。
摘要：
大规模视觉语言模型（LVLMs）能够处理图像和文本，在图像描述和描述生成等多模态任务中表现出色。然而，尽管这些模型擅长生成事实性内容，但它们在根据上下文生成和评估反映对同一图像不同视角的文本方面的能力尚未得到充分探索。为了解决这一问题，我们提出了IRR：图像评论排序（Image Review Rank），一种新颖的评估框架，旨在从多个视角评估评论性文本。IRR通过衡量LVLMs的判断与人类解释的一致性来评估它们。我们使用一个包含15个类别图像的数据集进行验证，每个类别有五条评论性文本及标注的排序，涵盖英语和日语，总计超过2,000个数据实例。数据集可在https://hf.co/datasets/naist-nlp/Wiki-ImageReview1.0获取。我们的结果表明，尽管LVLMs在不同语言间表现出一致性，但它们与人类标注的相关性不足，凸显了进一步改进的必要性。这些发现揭示了当前评估方法的局限性，以及需要更好地捕捉视觉与语言任务中人类推理的方法。
英文摘要：
Large-scale Vision-Language Models (LVLMs) process both images and text, excelling in multimodal tasks such as image captioning and description generation. However, while these models excel at generating factual content, their ability to generate and evaluate texts reflecting perspectives on the same image, depending on the context, has not been sufficiently explored. To address this, we propose IRR: Image Review Rank, a novel evaluation framework designed to assess critic review texts from multiple perspectives. IRR evaluates LVLMs by measuring how closely their judgments align with human interpretations. We validate it using a dataset of images from 15 categories, each with five critic review texts and annotated rankings in both English and Japanese, totaling over 2,000 data instances. The datasets are available at https://hf.co/datasets/naist-nlp/Wiki-ImageReview1.0. Our results indicate that, although LVLMs exhibited consistent performance across languages, their correlation with human annotations was insufficient, highlighting the need for further advancements. These findings highlight the limitations of current evaluation methods and the need for approaches that better capture human reasoning in Vision & Language tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
