Large Vision-Language Models (LVLMs) have exhibited remarkable progress. However, deficiencies remain compared to human intelligence, such as hallucination and shallow pattern matching. In this work, we aim to evaluate a fundamental yet underexplored intelligence: association, a cornerstone of human cognition for creative thinking and knowledge integration. Current benchmarks, often limited to closed-ended tasks, fail to capture the complexity of open-ended association reasoning vital for real-world applications. To address this, we present MM-OPERA, a systematic benchmark with 11,497 instances across two open-ended tasks: Remote-Item Association (RIA) and In-Context Association (ICA), aligning association intelligence evaluation with human psychometric principles. It challenges LVLMs to resemble the spirit of divergent thinking and convergent associative reasoning through free-form responses and explicit reasoning paths. We deploy tailored LLM-as-a-Judge strategies to evaluate open-ended outputs, applying process-reward-informed judgment to dissect reasoning with precision. Extensive empirical studies on state-of-the-art LVLMs, including sensitivity analysis of task instances, validity analysis of LLM-as-a-Judge strategies, and diversity analysis across abilities, domains, languages, cultures, etc., provide a comprehensive and nuanced understanding of the limitations of current LVLMs in associative reasoning, paving the way for more human-like and general-purpose AI. The dataset and code are available at https://github.com/MM-OPERA-Bench/MM-OPERA.
LEXam: Benchmarking Legal Reasoning on 340 Law Exams
ETH Zurich(苏黎世联邦理工学院);University of Zurich(苏黎世大学);University of Lausanne(洛桑大学);Max Planck Institute for Research on Collective Goods(马克斯·普朗克集体物品研究所);Omnilex;University of St. Gallen(圣加仑大学);Swiss Federal Supreme Court(瑞士联邦最高法院)
MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors
Department of Computer Science, ETH Zurich(苏黎世联邦理工学院计算机科学系);ETH AI Center(ETH人工智能中心);Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany(达姆施塔特技术大学计算机科学系);Professorship for Learning Sciences and Higher Education, ETH Zurich(苏黎世联邦理工学院学习科学与高等教育教授职位)
Benchmarking Foundation Models with Language-Model-as-an-Examiner
Singapore Management University(新加坡管理大学);Tsinghua University(清华大学);University of California, Los Angeles(加州大学洛杉矶分校);Beijing University of Posts and Telecommunications(北京邮电大学)
Benchmarking large language models for materials synthesis: the case of atomic layer deposition
Applied Materials Division, Argonne National Laboratory(阿贡国家实验室应用材料部门);Business and Information Systems, Argonne National Laboratory(阿贡国家实验室商业与信息系统部门);Northwestern Center for Water Research, Northwestern University(西北大学水研究中心)
Benchmarking the Thinking Mode of Multimodal Large Language Models in Clinical Tasks
Bytedance(字节跳动);Peking University(北京大学);The Chinese University of Hong Kong(香港中文大学);The University of Hong Kong(香港大学);Mohamed bin Zayed University of Artificial Intelligence(马尔代夫人工智能大学);Stanford University(斯坦福大学);University of Michigan(密歇根大学)