Grok、Deepseek Janus、Gemini、Qwen、Mistral 和 ChatGPT 的视觉推理评估
Visual Reasoning Evaluation of Grok, Deepseek Janus, Gemini, Qwen, Mistral, and ChatGPT
AI总结：
本研究提出一个结合多图像推理、弃权评估和熵指标的新基准，评估多个多模态大模型，发现 ChatGPT-o1 领先，并揭示模型规模非性能唯一决定因素。
摘要：
多模态大语言模型（LLMs）的传统评估受限于其对单图像推理的聚焦，未能评估诸如上下文理解、推理稳定性和不确定性校准等关键方面。本研究通过引入一个新颖的基准来弥补这些不足，该基准将多图像推理任务与基于弃权（不执行）的评估及位置偏差检测相结合。为评估这些维度，我们进一步引入熵作为量化跨重排答案变体推理一致性的新指标。我们将此基准应用于评估 Grok 3、ChatGPT-4o、ChatGPT-o1、Gemini 2.0 Flash Experimental、DeepSeek Janus 模型、Qwen2.5-VL-72B-Instruct、QVQ-72B-Preview 和 Pixtral 12B，涵盖八项视觉推理任务，包括差异识别和图表解读。我们的发现显示，ChatGPT-o1 在总体准确率（82.5%）和弃权（不执行）准确率（70.0%）上领先，紧随其后的是 Gemini 2.0 Flash Experimental（70.8%）。QVQ-72B-Preview 展现了卓越的弃权（不执行）准确率（85.5%）。值得注意的是，Pixtral 12B（51.7%）在特定领域显示出潜力，而 Janus 模型在偏差和不确定性校准方面表现出挑战，反映在较低的弃权（不执行）准确率和高熵分数上。Janus 模型的高熵分数（Janus 7B：0.8392，Janus 1B：0.787）凸显了其对位置偏差的易感性和不稳定的推理，与 ChatGPT 模型的低熵和稳健推理形成对比。研究进一步表明，模型规模并非性能的唯一决定因素，Grok 3 尽管参数数量庞大但表现不佳即为明证。通过采用多图像上下文、弃权（不执行）机制和基于熵的一致性指标，该基准为评估多模态 LLMs 设立了新标准，使得对下一代 AI 系统的评估更加稳健和可靠。
英文摘要：
Traditional evaluations of multimodal large language models (LLMs) have been limited by their focus on single-image reasoning, failing to assess crucial aspects like contextual understanding, reasoning stability, and uncertainty calibration. This study addresses these limitations by introducing a novel benchmark that integrates multi-image reasoning tasks with rejection-based evaluation and positional bias detection. To evaluate these dimensions, we further introduce entropy as a novel metric for quantifying reasoning consistency across reordered answer variants. We applied this benchmark to assess Grok 3, ChatGPT-4o, ChatGPT-o1, Gemini 2.0 Flash Experimental, DeepSeek Janus models, Qwen2.5-VL-72B-Instruct, QVQ-72B-Preview, and Pixtral 12B across eight visual reasoning tasks, including difference spotting and diagram interpretation. Our findings reveal ChatGPT-o1 leading in overall accuracy (82.5\%) and rejection accuracy (70.0\%), closely followed by Gemini 2.0 Flash Experimental (70.8\%). QVQ-72B-Preview demonstrated superior rejection accuracy (85.5\%). Notably, Pixtral 12B (51.7\%) showed promise in specific domains, while Janus models exhibited challenges in bias and uncertainty calibration, reflected in low rejection accuracies and high entropy scores. High entropy scores in Janus models (Janus 7B: 0.8392, Janus 1B: 0.787) underscore their susceptibility to positional bias and unstable reasoning, contrasting with the low entropy and robust reasoning of ChatGPT models. The study further demonstrates that model size is not the sole determinant of performance, as evidenced by Grok 3 underperformance despite its substantial parameter count. By employing multi-image contexts, rejection mechanisms, and entropy-based consistency metrics, this benchmark sets a new standard for evaluating multimodal LLMs, enabling a more robust and reliable assessment of next-generation AI systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
