Evaluating mathematical reasoning in LLMs is constrained by limited benchmark sizes and inherent model stochasticity, yielding high-variance accuracy estimates and unstable rankings across platforms. On difficult problems, an LLM may fail to produce a correct final answer, yet still provide reliable pairwise comparison signals indicating which of two candidate solutions is better. We leverage this observation to design a statistically efficient evaluation framework that combines standard labeled outcomes with pairwise comparison signals obtained by having models judge auxiliary reasoning chains. Treating these comparison signals as control variates, we develop a semiparametric estimator based on the efficient influence function (EIF) for the setting where auxiliary reasoning chains are observed. This yields a one-step estimator that achieves the semiparametric efficiency bound, guarantees strict variance reduction over naive sample averaging, and admits asymptotic normality for principled uncertainty quantification. Across simulations, our one-step estimator substantially improves ranking accuracy, with gains increasing as model output noise grows. Experiments on GPQA Diamond, AIME 2025, and GSM8K further demonstrate more precise performance estimation and more reliable model rankings, especially in small-sample regimes where conventional evaluation is pretty unstable.
A Comparative Study of Student Perspectives on Technical Writing Feedback Quality: Evaluating LLMs, SLMs, and Humans in Computer Science Topics
McMaster University（麦斯特大学）；Department of Computer Science, University of Toronto（多伦多大学计算机科学系）；Research Institute for the Study of University Pedagogy, University of Toronto Mississauga（多伦多大学密西根分校大学教学研究学院）；Department of Mathematical and Computational Sciences, University of Toronto Mississauga（多伦多大学密西根分校数学与计算科学系）
Evaluating Large Language Models on the Spanish Medical Intern Resident (MIR) Examination 2024/2025:A Comparative Analysis of Clinical Reasoning and Knowledge Application
Evaluating Large Language Models on Quantum Mechanics: A Comparative Study Across Diverse Models and Tasks
Quantum Science and Technology Laboratory, Physical Research Laboratory, Navrangpura, Ahmedabad 380009, India（量子科学与技术实验室，物理研究实验室，Ahmedabad）；Indian Institute of Technology Gandhinagar, Palaj, Gandhinagar 382055, India（印度理工学院冈达塞瓦尔）
The System Hallucination Scale (SHS): A Human-Centered Instrument for Assessing Perceived Reliability of Large Language Models
Machine Learning and Information Science Group, Medical University of Graz（格拉茨医科大学机器学习与信息科学组）；Human Machine Mind Cooperation, Graz, Austria（格拉茨人类机智合作中心）；MIDATA Cooperative, Zurich, Switzerland（苏黎世MIDATA合作组织）；Human-Centered AI Lab, BOKU University Vienna（维也纳BOKU大学人本AI实验室）
Fusing Knowledge and Language: A Comparative Study of Knowledge Graph-Based Question Answering with LLMs
Indian Institute of Technology, Kharagpur（印度理工学院卡拉格普尔分校）；Maharishi Markandeshwar (Deemed to be University)（马哈里希·马坎德什瓦尔大学）；IMD, Ministry of Earth Sciences（印度气象局，地球科学部）；NePeur（NePeur公司）
Comparative Separation: Evaluating Separation on Comparative Judgment Test Data
Department of Software Engineering, Rochester Institute of Technology（软件工程系，罗切斯特理工学院）；University of California, Santa Barbara（加州大学圣巴巴拉分校）；University of Vermont（佛罗里达大学）
Evaluating Named Entity Recognition: A comparative analysis of mono- and multilingual transformer models on a novel Brazilian corporate earnings call transcripts dataset
Instituto Federal de São Paulo - IFSP（圣保罗联邦教育、科学和技术研究所（IFSP））；Universidade Estadual de Campinas - Unicamp（坎皮纳斯州立大学（Unicamp））
University of Illinois at Urbana Champaign（伊利诺伊大学厄巴纳-香槟分校）
Reasoning is a fundamental capability for harnessing valuable insight, knowledge and patterns from knowledge graphs. Existing work has primarily been focusing on point-wise reasoning, including search, link predication, entity prediction, subgraph matching and so on. This paper introduces comparative reasoning over knowledge graphs, which aims to infer the commonality and inconsistency with respect to multiple pieces of clues. We envision that the comparative reasoning will complement and expand the existing point-wise reasoning over knowledge graphs. In detail, we develop KompaRe, the first o…
New York University（纽约大学）；University of Southern California（南加州大学）；HRI Europe GmbH（HRI欧洲有限公司）
Hanabi is a cooperative game that challenges exist-ing AI techniques due to its focus on modeling the mental states ofother players to interpret and predict their behavior. While thereare agents that can achieve near-perfect scores in the game byagreeing on some shared strategy, comparatively little progresshas been made in ad-hoc cooperation settings, where partnersand strategies are not known in advance. In this paper, we showthat agents trained through self-play using the popular RainbowDQN architecture fail to cooperate well with simple rule-basedagents that were not seen during training a…
Simulators make unique benchmarks for causal effect estimation since they do not rely on unverifiable assumptions or the ability to intervene on real-world systems, but are often too simple to capture important aspects of real applications. We propose a simulator of Alzheimer's disease aimed at modeling intricacies of healthcare data while enabling benchmarking of causal effect and policy estimators. We fit the system to the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset and ground hand-crafted components in results from comparative treatment trials and observational treatment patt…
