面向视觉语言模型评估的挑战性多项选择题自动生成
Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation
AI总结：
本文提出智能体框架AutoConverter，将开放式VQA问题自动转换为多项选择题，并构建包含9,018道题的VMCBench基准，用于对33个视觉语言模型进行可扩展且可复现的客观评估。
摘要：
视觉语言模型（VLMs）的快速发展要求进行严格且可靠的评估。然而，当前的视觉问答（VQA）基准通常依赖于开放式问题，由于自然语言回答的变异性，使得准确评估变得困难。为了解决这一问题，我们提出了AutoConverter，这是一个智能体框架，能够自动将这些开放式问题转换为多项选择格式，从而实现客观评估，同时降低昂贵的多项选择题创建过程。我们的实验表明，AutoConverter能够生成正确且具有挑战性的多项选择题，与人工创建的问题相比，VLMs在这些问题上的准确率表现出一致地相似或更低。利用AutoConverter，我们构建了VMCBench，这是一个通过将20个现有VQA数据集转换为统一多项选择格式而创建的基准，共计9,018个问题。我们在VMCBench上全面评估了33个最先进的VLMs，为可扩展、一致且可复现的VLM评估设立了新的标准。
英文摘要：
The rapid development of vision language models (VLMs) demands rigorous and reliable evaluation. However, current visual question answering (VQA) benchmarks often depend on open-ended questions, making accurate evaluation difficult due to the variability in natural language responses. To address this, we introduce AutoConverter, an agentic framework that automatically converts these open-ended questions into multiple-choice format, enabling objective evaluation while reducing the costly multiple-choice question creation process. Our experiments demonstrate that AutoConverter can generate correct and challenging multiple-choice questions, with VLMs demonstrating consistently similar or lower accuracy on these questions compared to human-created ones. Using AutoConverter, we construct VMCBench, a benchmark created by transforming 20 existing VQA datasets into a unified multiple-choice format, totaling 9,018 questions. We comprehensively evaluate 33 state-of-the-art VLMs on VMCBench, setting a new standard for scalable, consistent, and reproducible VLM evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
