arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.26937cs.LG
收藏

MM-OPERA:大型视觉语言模型的开放式联想推理基准测试

MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models

Sun Yat-sen University(中山大学);Peng Cheng Laboratory(鹏城实验室);Jinan University(暨南大学)

Zimeng Huang, Jinxin Ke, Xiaoxuan Fan, Yufeng Yang, Yang Liu, Liu Zhonghan, Zedi Wang, Junteng Dai, Haoyi Jiang, Yuyu Zhou, Keze Wang, Ziliang Chen

AI总结:

本文提出MM-OPERA基准,通过RIA和ICA两项开放式任务评估大型视觉语言模型的联想推理能力,发现当前模型在类人认知方面存在局限。

摘要:

大型视觉语言模型(LVLMs)已展现出显著进展。然而,与人类智能相比仍存在不足,如幻觉和浅层模式匹配。在这项工作中,我们旨在评估一种基础但尚未充分探索的智能:联想,这是人类创造性思维和知识整合的基石。当前的基准测试通常局限于封闭式任务,无法捕捉对现实世界应用至关重要的开放式联想推理的复杂性。为此,我们提出了MM-OPERA,这是一个包含11,497个实例的系统化基准,涵盖两项开放式任务:远距项目联想(RIA)和上下文联想(ICA),将联想智能评估与人类心理测量原则对齐。它通过自由形式回答和显式推理路径,挑战LVLMs以体现发散思维和聚合联想推理的精神。我们部署了定制的LLM-as-a-Judge策略来评估开放式输出,应用过程奖励引导的判断来精准剖析推理。对最先进LVLMs的广泛实证研究,包括任务实例敏感性分析、LLM-as-a-Judge策略有效性分析,以及跨能力、领域、语言、文化等的多样性分析,为当前LVLMs在联想推理上的局限性提供了全面而细致的理解,为更类人、更通用的AI铺平道路。数据集和代码可在https://github.com/MM-OPERA-Bench/MM-OPERA获取。

英文摘要:

Large Vision-Language Models (LVLMs) have exhibited remarkable progress. However, deficiencies remain compared to human intelligence, such as hallucination and shallow pattern matching. In this work, we aim to evaluate a fundamental yet underexplored intelligence: association, a cornerstone of human cognition for creative thinking and knowledge integration. Current benchmarks, often limited to closed-ended tasks, fail to capture the complexity of open-ended association reasoning vital for real-world applications. To address this, we present MM-OPERA, a systematic benchmark with 11,497 instances across two open-ended tasks: Remote-Item Association (RIA) and In-Context Association (ICA), aligning association intelligence evaluation with human psychometric principles. It challenges LVLMs to resemble the spirit of divergent thinking and convergent associative reasoning through free-form responses and explicit reasoning paths. We deploy tailored LLM-as-a-Judge strategies to evaluate open-ended outputs, applying process-reward-informed judgment to dissect reasoning with precision. Extensive empirical studies on state-of-the-art LVLMs, including sensitivity analysis of task instances, validity analysis of LLM-as-a-Judge strategies, and diversity analysis across abilities, domains, languages, cultures, etc., provide a comprehensive and nuanced understanding of the limitations of current LVLMs in associative reasoning, paving the way for more human-like and general-purpose AI. The dataset and code are available at https://github.com/MM-OPERA-Bench/MM-OPERA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑