发表机构
The Hong Kong Polytechnic University; Eastern Institute of Technology; Harbin Institute of Technology (Shenzhen)（香港理工大学; 东方理工大学; 哈尔滨工业大学（深圳））
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出MMOOC基准，评估多模态大语言模型在上下文偏移场景下的拒绝与作答能力，实验发现当前模型难以平衡可答性与拒绝性，后训练可提升稳健性，该基准将公开。
AI 中文摘要
多模态大语言模型（Multimodal Large Language Models, MLLMs）在各类视觉-语言任务上已取得优异性能，但在不完美或偏移的上下文环境中常出现失效情况。可靠的MLLM应在遭遇主题级上下文偏移的真正上下文外（Out-of-Context, OOC）问题时拒绝回答，而在遭遇非主题上下文偏移的偏移上下文内（Shifted In-Context, Shifted IC）可回答问题时仍能正确作答。现有基准主要针对OOC或视觉无法回答的问题，却忽略了可回答的Shifted IC案例，且覆盖的OOC偏移类型有限。为填补这一空白，本文提出MMOOC——一个用于评估MLLM拒绝回答与稳健作答能力的大规模基准。MMOOC包含超过4.1万组图像-问题对，涵盖可回答的Shifted IC案例与无法回答的OOC案例，涉及三种问题格式、八种偏移类型及六种视觉场景，其数据质量通过基于MLLM的筛选与人工验证得到保障。我们采用准确率与拒绝率评估模型响应，并引入LLM作为评判者（LLM-as-a-Judge）指标评估模型推理的正确性。对不同MLLMs的实验表明，当前模型仍难以在偏移上下文环境中平衡可答性与拒绝性；我们进一步分析了关键失败模式，并发现后训练可提升模型稳健性。MMOOC将公开提供。
英文摘要
Multimodal Large Language Models (MLLMs) have achieved strong performance on a wide range of vision-language tasks, but often fail under imperfect or shifted contexts. A reliable MLLM should refuse truly out-of-context (OOC) questions with subject-level context shifts while still answering shifted in-context (Shifted IC) questions with non-subject context shifts. Existing benchmarks mainly target OOC or visually unanswerable questions, but overlook answerable Shifted IC cases and cover limited OOC shifts. To fill this gap, we present MMOOC, a large-scale benchmark for evaluating refusal and robust answering abilities of MLLMs. MMOOC contains over 41K image-question pairs, including answerable Shifted IC cases and unanswerable OOC cases, spanning three question formats, eight shift types and six visual scenarios, with data quality ensured through MLLM-based filtering and human verification. We evaluate model responses using Accuracy and Refusal Rate, and further introduce an LLM-as-a-Judge metric to assess the correctness of model reasoning. Experiments on diverse MLLMs show that current models still struggle to balance answer-ability and refusal under shifted contexts. We further analyze key failure patterns and show that post-training can improve robustness. MMOOC will be made publicly available.
Commentsproject page:https://zhuwenjie98.github.io/MMOOC-project-page/
