Using Multimodal Large Language Models (MLLMs) as judges to achieve precise and consistent evaluations has gradually become an emerging paradigm across various domains. Evaluating the capability and reliability of MLLM-as-a-judge systems is therefore essential for ensuring trustworthy assessment. Existing judge benchmarks categorize samples by task types but fail to capture the fundamental judgment capabilities required for reliable evaluation. In this work, we introduce M-JudgeBench, a ten-dimensional capability-oriented benchmark designed to comprehensively assess the judgment abilities of MLLMs. Our benchmark decomposes evaluation into pairwise Chain-of-Thought (CoT) comparison, length bias avoidance, and process error detection tasks, jointly covering ten fine-grained subtasks. This design enables diagnosis of model reliability across reasoning styles, response lengths, and cross-model variations. Systematic evaluation uncovers the systematic weaknesses in existing MLLM-as-a-judge systems. To address this issue, we further propose Judge-MCTS, a data construction framework generating pairwise reasoning trajectories with various correctness and length. Using Judge-MCTS, we construct an MCTS-augmented dataset and train M-Judger, a series of strong judge models. Extensive experiments demonstrate the superiority of M-Judger on existing judge benchmarks as well as M-JudgeBench. Overall, our work establishes a more principled foundation for evaluating MLLM-as-a-judge through M-JudgeBench and Judge-MCTS framework, paving the way for future research on judge model evaluation and capability-driven judge training.
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
The University of Hong Kong（香港大学）；Nanjing University（南京大学）；University of Science and Technology of China（中国科学技术大学）；National University of Singapore（新加坡国立大学）；Fudan University（复旦大学）
WiFiPenTester: Advancing Wireless Ethical Hacking with Governed GenAI
Diwan of Royal Court, Muscat, Oman.（阿曼穆斯cat王室法庭）；Royal Holloway, University of London, Egham, UK.（伦敦皇家霍洛威大学）；German University of Technology in Oman, Muscat, Oman.（阿曼技术大学）
WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation
Mininglamp Technology（旷视科技）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所综合技术集成中心）
MMLab, The Chinese University of Hong Kong（中大香港实验室）；CPII under InnoHK（创新香港 CPII）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Loop Area Institute（深圳河套学院）；Shandong University（山东大学）；Huawei Technologies（华为技术）
Hybrid Retrieval-Augmented Generation Agent for Trustworthy Legal Question Answering in Judicial Forensics
Department of Electronic Engineering, City University of Hong Kong(DongGuan)（香港城市大学（东莞）电子工程系）；Department of Electronic Engineering, City University of Hong Kong（香港城市大学电子工程系）
VectorGym: A Multi-Task Benchmark for SVG Code Generation, Sketching and Editing
ServiceNow Research（ServiceNow 研究院）；Mila, Quebec AI Institute（Mila 魁北克人工智能研究所）；Columbia University（哥伦比亚大学）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Stony Brook University（石溪大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Minzu University of China（中央民族大学）；University of Waterloo（滑铁卢大学）；Canada CIFAR AI Chair（加拿大 CIFAR 人工智能讲席）；Computer Vision Center（计算机视觉中心）
ORCA: Open-ended Response Correctness Assessment for Audio Question Answering
Brno University of Technology（布尔诺科技大学）；Universidad Autónoma de Madrid（马德里自治大学）；University of Buenos Aires（布宜诺斯艾利斯大学）；Tufts University（塔夫茨大学）；University of Maryland（马里兰大学）
