arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.06591cs.CL
收藏

Prometheus-Vision:作为细粒度评估裁判的视觉-语言模型

Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation

KAIST AI(韩国科学技术院人工智能研究所);NAVER AI Lab(NAVER人工智能实验室);NAVER Cloud AI(NAVER云人工智能)

Seongyun Lee, Seungone Kim, Sue Hyun Park, Geewook Kim, Minjoon Seo

AI总结:

本文提出用VLM评估VLM的框架,构建包含15K定制评分标准的Perception Collection数据集,并训练首个开源VLM评估器Prometheus-Vision,其与人类及GPT-4V的相关性在开源模型中最高。

摘要:

评估视觉-语言模型(VLMs)生成的长篇回答具有挑战性。这不仅需要检查VLM是否遵循给定指令,还需要验证文本输出是否恰当地基于给定图像。受近期用语言模型评估语言模型的方法启发,本文提出用VLM来评估VLM。为此,我们发布了一个名为Perception Collection的新反馈数据集,包含15K条用户在评估时可能关心的定制评分标准。利用Perception Collection,我们训练了Prometheus-Vision,这是首个能够在评估过程中理解用户自定义评分标准的开源VLM评估器模型。在开源模型中,Prometheus-Vision与人类评估者和GPT-4V的Pearson相关性最高,证明了其在透明且可及的VLM评估方面的有效性。我们在https://github.com/kaistAI/prometheus-vision开源了代码、数据集和模型。

英文摘要:

Assessing long-form responses generated by Vision-Language Models (VLMs) is challenging. It not only requires checking whether the VLM follows the given instruction but also verifying whether the text output is properly grounded on the given image. Inspired by the recent approach of evaluating LMs with LMs, in this work, we propose to evaluate VLMs with VLMs. For this purpose, we present a new feedback dataset called the Perception Collection, encompassing 15K customized score rubrics that users might care about during assessment. Using the Perception Collection, we train Prometheus-Vision, the first open-source VLM evaluator model that can understand the user-defined score criteria during evaluation. Prometheus-Vision shows the highest Pearson correlation with human evaluators and GPT-4V among open-source models, showing its effectiveness for transparent and accessible evaluation of VLMs. We open-source our code, dataset, and model at https://github.com/kaistAI/prometheus-vision

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑