MMPCBench：评估多模态大语言模型对有缺陷输入的主动批判能力的基准

中文摘要 AI 辅助

随着多模态大语言模型（MLLMs）发展为复杂的交互式助手，其可靠性不仅取决于遵循指令，还取决于对指令的验证。我们将主动批判定义为模型无需额外提示即可自主识别、分析并修复有缺陷用户输入的能力。然而，现有评估主要在理想环境或简单拒绝行为下测试模型，很大程度上忽略了主动错误处理。为填补这一空白，我们提出MMPCBench，一个用于评估MLLMs主动批判能力的综合框架。它包含细粒度分类体系，涵盖4种主要错误类型及12个子类别，范围从跨模态矛盾到缺失视觉前提。我们采用分层评估协议，衡量模型的错误检测、诊断与解决性能，并应用感知对齐的指标评估内部推理与最终响应的一致性。对14种主流MLLMs的测试显示，其在主动批判方面存在明显弱点，尤其在处理细微视觉异常时。值得注意的是，我们发现普遍存在的“一致性缺口”：推理模型常能在内部推理中正确识别并分析错误，但会在最终输出中抑制这些有效见解，以优先保证响应合规性。代码和数据可在此httpsURL获取。

英文摘要

As Multimodal Large Language Models (MLLMs) evolve into sophisticated interactive assistants, their reliability depends not only on following instructions but also on validating them. We define Proactive Critique as the model's autonomous ability to identify, analyze and fix faulty user inputs without extra prompts. However, evaluations mainly test models under ideal circumstances or simple refusal behaviors, largely ignoring active error processing. To fill this gap, we propose MMPCBench, a comprehensive framework for evaluating MLLMs' proactive critique competence. It features a fine-grained taxonomy of 4 primary error types spanning 12 subcategories, ranging from cross-modal contradictions to missing visual premises. We adopt a hierarchical evaluation protocol to measure models' error detection, diagnosis and resolution performance, and apply alignment-aware metrics to assess the coherence between internal reasoning and final responses. Tests on 14 mainstream MLLMs show obvious weaknesses in proactive critique, especially in dealing with subtle visual anomalies. Notably, we identify a pervasive "consistency gap": reasoning models can often correctly identify and analyze errors during internal reasoning yet suppress these valid insights in final outputs to prioritize response compliance. The code and data is available at https://github.com/ALIENS32/MMPCBench.

发表机构

• School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）
• International Center of Future Science, Jilin University（吉林大学未来科学国际中心）
• Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China（教育部知识驱动人机智能工程研究中心）

机构由 AI 辅助整理，请以论文原文为准。