评估视觉问答基准中的方差
Evaluating Variance in Visual Question Answering Benchmarks
AI总结：
本文分析14个VQA基准中的性能方差，研究训练种子、框架非确定性、模型规模和指令微调的影响，并探索完形填空式评估以减少随机性，倡导方差感知的评估方法。
摘要：
多模态大语言模型（MLLMs）已成为视觉问答（VQA）的强大工具，能够跨视觉和文本模态进行推理和上下文理解。尽管取得了进展，但在VQA基准上对MLLMs的评估通常依赖于点估计，忽视了由随机模型输出、训练种子敏感性和超参数配置等因素引起的性能显著方差。本文通过分析14个广泛使用的VQA基准中的方差来严格审视这些问题，涵盖视觉推理、文本理解和常识推理等多样任务。我们系统研究了训练种子、框架非确定性、模型规模和扩展指令微调对性能变异性的影响。此外，我们探索了完形填空式评估作为替代评估策略，研究其在减少随机性和提高跨基准可靠性方面的有效性。我们的发现凸显了当前评估实践的局限性，并倡导采用方差感知方法以促进MLLMs更稳健和可靠的发展。
英文摘要：
Multimodal large language models (MLLMs) have emerged as powerful tools for visual question answering (VQA), enabling reasoning and contextual understanding across visual and textual modalities. Despite their advancements, the evaluation of MLLMs on VQA benchmarks often relies on point estimates, overlooking the significant variance in performance caused by factors such as stochastic model outputs, training seed sensitivity, and hyperparameter configurations. This paper critically examines these issues by analyzing variance across 14 widely used VQA benchmarks, covering diverse tasks such as visual reasoning, text understanding, and commonsense reasoning. We systematically study the impact of training seed, framework non-determinism, model scale, and extended instruction finetuning on performance variability. Additionally, we explore Cloze-style evaluation as an alternate assessment strategy, studying its effectiveness in reducing stochasticity and improving reliability across benchmarks. Our findings highlight the limitations of current evaluation practices and advocate for variance-aware methodologies to foster more robust and reliable development of MLLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
