InfiMM-Eval:面向多模态大语言模型的复杂开放式推理评估
InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models
AI总结:
本文提出InfiMM-Eval基准,通过演绎、溯因和类比三类复杂开放式推理任务,并结合中间推理步骤评估多模态大语言模型的推理能力,弥补现有基准仅关注简单问答的不足。
摘要:
多模态大语言模型(MLLMs)在人工智能领域日益突出。这些模型不仅在传统的视觉-语言任务中表现出色,而且在当代多模态基准测试中也展现出令人印象深刻的性能。尽管许多此类基准测试试图全面评估MLLMs,但它们通常集中于基础推理任务,往往只产生简单的是/否或多选回答。这些方法自然会导致在确定MLLMs推理能力时产生混淆和困难。为了缓解这一问题,我们手动构建了一个专门为MLLMs设计的基准数据集,重点关注复杂推理任务。我们的基准包含三个关键推理类别:演绎推理、溯因推理和类比推理。我们数据集中的查询经过有意构造,以使MLLMs在生成答案的过程中调动其推理能力。为了在各种MLLMs之间进行公平比较,我们将中间推理步骤纳入评估标准。在MLLM无法给出确定答案的情况下,通过要求其提供中间推理步骤来评估其推理能力。如果这些步骤与我们的手动标注一致,则给予相应分数。这种评估方案类似于人类评估中常用的方法,例如考试或作业,并且我们认为与现有基准相比,这是一种更有效的评估技术。我们使用这一经过严格开发的开放式多步精细推理基准评估了一组具有代表性的MLLMs,该基准旨在挑战并准确衡量它们的推理能力。代码和数据将在https://infimm.github.io/InfiMM-Eval/发布。
英文摘要:
Multi-modal Large Language Models (MLLMs) are increasingly prominent in the field of artificial intelligence. These models not only excel in traditional vision-language tasks but also demonstrate impressive performance in contemporary multi-modal benchmarks. Although many of these benchmarks attempt to holistically evaluate MLLMs, they typically concentrate on basic reasoning tasks, often yielding only simple yes/no or multi-choice responses. These methods naturally lead to confusion and difficulties in conclusively determining the reasoning capabilities of MLLMs. To mitigate this issue, we manually curate a benchmark dataset specifically designed for MLLMs, with a focus on complex reasoning tasks. Our benchmark comprises three key reasoning categories: deductive, abductive, and analogical reasoning. The queries in our dataset are intentionally constructed to engage the reasoning capabilities of MLLMs in the process of generating answers. For a fair comparison across various MLLMs, we incorporate intermediate reasoning steps into our evaluation criteria. In instances where an MLLM is unable to produce a definitive answer, its reasoning ability is evaluated by requesting intermediate reasoning steps. If these steps align with our manual annotations, appropriate scores are assigned. This evaluation scheme resembles methods commonly used in human assessments, such as exams or assignments, and represents what we consider a more effective assessment technique compared with existing benchmarks. We evaluate a selection of representative MLLMs using this rigorously developed open-ended multi-step elaborate reasoning benchmark, designed to challenge and accurately measure their reasoning capabilities. The code and data will be released at https://infimm.github.io/InfiMM-Eval/
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。