KOFFVQA：面向韩语大型视觉语言模型的客观评估自由形式VQA基准
KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large Vision-Language Models in the Korean Language
AI总结：
针对韩语VLMs评估缺乏且现有方法主观不可靠的问题，提出KOFFVQA韩语自由形式视觉问答基准，通过预定义客观评分标准实现可靠评估，实验证明其优于现有方法。
摘要：
近期大型视觉语言模型（VLMs）的出现催生了多种用于评估此类模型的不同基准。尽管如此，我们观察到大多数现有评估方法都存在一个事实问题：它们要么要求模型从预先确定的回答中进行选择，从而牺牲了开放性；要么使用评判模型来评估回答，导致评估主观且不可靠。此外，我们观察到缺乏针对韩语VLMs的基准，而这是作为更常见的英语基准之外的独立指标所必需的，因为生成式语言模型的性能可能会因所用语言的不同而存在显著差异。因此，我们提出了KOFFVQA，这是一个用于评估VLMs的韩语通用自由形式视觉问答基准。我们的基准包含275个精心设计的问题，每个问题都配有图像和涵盖VLM性能10个不同方面的评分标准。这些评分标准允许评判模型基于预先确定的规则集对每个回答进行评分，从而消除了不可靠的问题。通过以客观方式定义评估标准，即使是小型开源模型也能可靠地用于在我们的基准上评估模型。除了在我们的基准上评估大量现有VLMs外，我们还通过实验验证，我们这种使用预先存在的评分标准进行评估的方法比现有方法可靠得多。我们的评估代码可在 https://github.com/maum-ai/KOFFVQA 获取。
英文摘要：
The recent emergence of Large Vision-Language Models(VLMs) has resulted in a variety of different benchmarks for evaluating such models. Despite this, we observe that most existing evaluation methods suffer from the fact that they either require the model to choose from pre-determined responses, sacrificing open-endedness, or evaluate responses using a judge model, resulting in subjective and unreliable evaluation. In addition, we observe a lack of benchmarks for VLMs in the Korean language, which are necessary as a separate metric from more common English language benchmarks, as the performance of generative language models can differ significantly based on the language being used. Therefore, we present KOFFVQA, a general-purpose free-form visual question answering benchmark in the Korean language for the evaluation of VLMs. Our benchmark consists of 275 carefully crafted questions each paired with an image and grading criteria covering 10 different aspects of VLM performance. The grading criteria eliminate the problem of unreliability by allowing the judge model to grade each response based on a pre-determined set of rules. By defining the evaluation criteria in an objective manner, even a small open-source model can be used to evaluate models on our benchmark reliably. In addition to evaluating a large number of existing VLMs on our benchmark, we also experimentally verify that our method of using pre-existing grading criteria for evaluation is much more reliable than existing methods. Our evaluation code is available at https://github.com/maum-ai/KOFFVQA
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
