How Many Code and Test Cases Are Enough? Evaluating Test Cases Generation from a Binary-Matrix Perspective
Harbin Institute of Technology（哈尔滨工业大学）；Central South University（中南大学）；Chinese Academy of Sciences（中国科学院）；Peking University（北京大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
Evaluating test cases automatically generated by Large Language Models (LLMs) is a critical yet challenging task. Existing benchmarks often evaluate the exclusion ratio on large, unstructured collections of wrong codes, suffering from high computational costs and score inflation. Furthermore, they inadvertently reward generators that detect common, trivial bugs, while failing to penalize their inability to identify rare yet critical faults. In this work, we connect two fundamental questions: (1) What is the minimal set of wrong codes sufficient to represent the entire error space? and (2) What is the minimal set of test cases needed to distinguish them? We introduce a novel framework that formalizes benchmark construction as finding an optimal diagnostic basis in a binary code-test matrix, where rows represent wrong codes and columns represent test case results. The rank of this matrix specifies the minimal number of independent error patterns (wrong codes) and provides a tight upper bound on the number of test cases required for complete fault coverage. Our objective is to identify a basis of size equal to the matrix rank that maximizes internal diversity. To tackle this NP-hard problem, we propose WrongSelect, an efficient approximation algorithm to select maximally diverse wrong codes. Applying this framework to millions of competitive programming submissions, we construct TC-Bench, a compact, diverse, and inflation-resistant benchmark. Extensive experiments show that even the most advanced test case generation methods achieve only ~60% exclusion rates on TC-Bench, exposing a significant gap in their diagnostic power and highlighting substantial room for future improvement. Our dataset is available at: https://huggingface.co/datasets/Luoberta/TC-Bench and our code is at: https://github.com/Luowaterbi/TC-Bench.
Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations
Faculty of Computer Science, UniVie Doctoral School Computer Science（计算机科学系，维也纳大学计算机科学博士学院）；Faculty of Philological and Cultural Studies, University of Vienna, Austria（文学与文化研究系，维也纳大学，奥地利）；ILLC, University of Amsterdam, Netherlands（阿姆斯特丹大学ILLC，荷兰）；Khoury College of Computer Sciences, Northeastern University, USA（东北大学计算机科学学院，美国）；LMU Munich, Munich Center for Machine Learning (MCML), Germany（慕尼黑大学，慕尼黑机器学习中心（MCML），德国）
Evaluating Personalized Tool-Augmented LLMs from the Perspectives of Personalization and Proactivity
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Evaluating Large Language Model Creativity from a Literary Perspective
Imperial College London（帝国理工学院）；Google DeepMind（谷歌DeepMind）；Institute of Historical Research, School of Advanced Study, University of London（伦敦大学高级研究院历史研究所）；University of Southampton（南安普顿大学）
Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院人工智能安全国家重点实验室，计算技术研究所）；Key Laboratory of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院网络数据科学与技术重点实验室，计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspective
Westlake University（西湖大学）；Westlake Institute for Advanced Study（西湖高等研究院）；Central South University（中南大学）；University of Electronic Science and Technology of China（电子科技大学）；Microsoft Research Asia（微软亚洲研究院）
Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from Psychometrics
School of Architecture, Tsinghua University（清华大学建筑学院）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；BNRist, Tsinghua University（清华大学BNRist）
STEER-BENCH: A Benchmark for Evaluating the Steerability of Large Language Models
Department of Computer Science, University of Southern California（计算机科学系，南加州大学）；Information Sciences Institute, University of Southern California（信息科学研究所，南加州大学）
WorldView-Bench: A Benchmark for Evaluating Global Cultural Perspectives in Large Language Models
Information Technology University Department of Computer Science（信息科技大学计算机科学系）；Zayed University College of Interdisciplinary Studies（扎耶德大学跨学科研究学院）；Computer Science and Engineering Department（计算机科学与工程系）
EmotionQueen: A Benchmark for Evaluating Empathy of Large Language Models
Fudan University（复旦大学）；School of Computer Science, Fudan University（复旦大学计算机科学学院）；Beijing Yuandian Technology（北京原点科技）；School of Engineering and Applied Science, University of Pennsylvania（宾夕法尼亚大学工程与应用科学学院）
EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models
Tsinghua University（清华大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；University of Toronto（多伦多大学）；Zhili College, Tsinghua University（清华大学至理书院）
SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactions
The University of Hong Kong（香港大学）；Carnegie Mellon University（卡内基梅隆大学）；Johns Hopkins University（约翰霍普金斯大学）；University of California Irvine（加州大学尔湾分校）；Stanford University（斯坦福大学）
A Multi-Perspective Benchmark and Moderation Model for Evaluating Safety and Adversarial Robustness
Department of Computer and Information Science, Fordham University（福德汉大学计算机与信息科学系）；College of Science and Engineering, Hamad Bin Khalifa University（哈马德·本·卡西姆大学科学与工程学院）；Department of Computer Science and Engineering, Qatar University（卡塔尔大学计算机科学与工程系）
Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective
Fudan University（复旦大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Zhejiang University（浙江大学）；Renmin University of China（中国人民大学）；Alibaba Group（阿里巴巴集团）
