BYO-Eval：为多模态语言模型的细粒度视觉评估构建你自己的数据集
BYO-Eval: Build Your Own Dataset for Fine-Grained Visual Assessment of Multimodal Language Models
AI总结：
本文提出BYO-Eval评估方法，通过程序化生成难度渐进的合成图像，对视觉语言模型进行细粒度感知失败诊断，从而替代高成本且易泄露信息的传统基准测试。
摘要：
视觉语言模型（VLM）现已足够先进，能够支持广泛的应用，包括回答复杂的视觉问题，并且越来越被期望以多种方式与图像进行交互。为了评估它们，当前的基准测试通常聚焦于特定领域（例如阅读图表），构建带有预定义多项选择题（MCQ）的标注真实图像数据集，以报告总体准确率分数。然而，此类基准测试标注成本高，存在信息泄露风险，并且无法阐明失败是源于视觉感知、推理还是通用知识的局限。我们提出一种受眼科诊断启发的新评估方法，利用合成图像的程序化生成来获得对视觉属性的控制，并精确揭示VLM中的感知失败。具体而言，我们构建图像集合，在保持其他视觉参数不变的情况下，逐步增加感兴趣内容的变化难度（例如计数任务中的物体数量）。这种诊断方法允许系统性的压力测试和细粒度失败分析，将重点从粗略的基准测试转向对VLM能力的有针对性和可解释的评估。我们的代码可在 https://github.com/byoeval/BYO-EVAL 获取。
英文摘要：
Visual Language Models (VLMs) are now sufficiently advanced to support a broad range of applications, including answering complex visual questions, and are increasingly expected to interact with images in varied ways. To evaluate them, current benchmarks often focus on specific domains (e.g., reading charts), constructing datasets of annotated real images paired with pre-defined Multiple Choice Questions (MCQs) to report aggregate accuracy scores. However, such benchmarks entail high annotation costs, risk information leakage, and do not clarify whether failures stem from limitations in visual perception, reasoning, or general knowledge. We propose a new evaluation methodology, inspired by ophthalmologic diagnostics, leveraging procedural generation of synthetic images to obtain control over visual attributes and precisely reveal perception failures in VLMs. Specifically, we build collections of images with gradually more challenging variations in the content of interest (e.g., number of objects in a counting task) while holding other visual parameters constant. This diagnostic allows systematic stress testing and fine-grained failure analysis, shifting the focus from coarse benchmarking toward targeted and interpretable assessment of VLM capabilities. Our code is available at https://github.com/byoeval/BYO-EVAL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
