The Illusion of AI Expertise Under Uncertainty: A Probabilistic Paradigm to Interpret Results Under Ground Truth Variation
Independent Researcher(独立研究员);Amazon(亚马逊);Gazi University(加齐大学);University of Colorado(科罗拉多大学);Samsun University(萨姆松大学);The University Of Melbourne(墨尔本大学);New York University(纽约大学);RMIT University(皇家墨尔本理工大学);University of Pennsylvania(宾夕法尼亚大学)
Aparna Elangovan, Lei Xu, Mahsa Elyasi, Ismail Akdulum, Mehmet Aksakal, Enes Gurun, Brian Hur, Saab Mansour, Ravid Shwartz Ziv, Karin Verspoor, Dan Roth
Benchmarking the capabilities of AI systems, including Large Language Models (LLMs) and Vision Models, typically ignores the impact of uncertainty in the underlying ground truth answers from experts. This ambiguity is not just limited to human preferences, but is also consequential even in safety critical domains such as medicine where uncertainty is pervasive. In this paper, we introduce a probabilistic paradigm to theoretically explain how high certainty in ground truth answers is almost always necessary for even an expert to achieve high scores. In contrast, in datasets with high variation in ground truth answers, the expert performance drops substantially resulting in almost no difference between a random labeller and an expert. This characteristic also manifests when comparing models, where uncertainty obfuscates differences between poor and high performing models. Therefore, ignoring uncertainty in ground truth evaluation data can result in the misleading conclusion that a non-expert has similar performance to that of an expert. Given ground truth answer variability and commonly used F1 and accuracy metrics, we bring forth a simple and explainable probabilistic coin toss model to estimate expected accuracyand expected F1 and compare the estimated scores with those that experts (human or model) achieve across multiple domains -- including radiology and models such as GPT-5 and Gemini-3.
The results lead to the recommendation that stratification by the probability of the ground truth answer becomes critical when expert performance is relatively low. Under stratified evaluation, performance comparison becomes more reliable in high certainty bins, mitigating the effect of the key confounding factor --- uncertainty.
Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilities
Adelaide University(阿德莱德大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Tsinghua University(清华大学);Amap, Alibaba Group(阿里巴巴集团高德地图);Beihang University(北京航空航天大学)
From Physician Expertise to Clinical Agents: Preserving, Standardizing, and Scaling Physicians' Medical Expertise with Lightweight LLM
the School of Chinese Medicine, the Beijing University of Chinese Medicine, Beijing, China(中国中医药大学中医学院,北京中医药大学,北京,中国);the School of Pharmacy, Nanjing University of Chinese Medicine, Nanjing, China(南京中医药大学药学院,南京,中国);the Infectious disease department, Dongfang Hospital, Beijing University of Chinese Medicine, Beijing, China(北京中医药大学东四医院感染科,北京,中国);the Gulou Hospital of Traditional Chinese Medicine of Beijing,Beijing, China(北京中医药大学附属鼓楼中医医院,北京,中国);the Department of Computer Science, Johns Hopkins University, Baltimore, USA(约翰霍普金斯大学计算机科学系,美国马里兰州巴尔的摩市);the Department of Education, Dongzhimen Hospital, Beijing University of Chinese Medicine, Beijing, China(北京中医药大学东直门医院教育部,北京,中国);the Department of Pediatrics, Wangjing Hospital, China Academy of Chinese Medical Sciences, Beijing, China(中国医学科学院北京协和医院儿科部,北京,中国);the School of Information Engineering, Huzhou University, Huzhou, China(湖州大学信息工程学院,湖州,中国);Research Center for Scientific Data Hub, Zhejiang Lab, Hangzhou, China(浙江省实验室科学数据中心研究中心,杭州,中国);the Frontier Basic Research Center, Zhejiang Lab, Hangzhou, China(浙江省实验室前沿基础研究中心,杭州,中国);the Research Center for High Efficiency Computing Infrastructure, Zhejiang Lab, Hangzhou, China(浙江省实验室高效计算基础设施研究中心,杭州,中国)
The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation
University of Toronto Computer Science(多伦多大学计算机科学系);University of Toronto Engineering Science(多伦多大学工程科学系);Carnegie Mellon University School of Computer Science(卡内基梅隆大学计算机科学学院);Yale University American Studies(耶鲁大学美国研究系);University of Waterloo Computer Science(滑铁卢大学计算机科学系);Google Research(谷歌研究);University of Toronto(多伦多大学);Carnegie Mellon University(卡内基梅隆大学);Yale University(耶鲁大学);University of Waterloo(滑铁卢大学)
Transparency as Architecture: Structural Compliance Gaps in EU AI Act Article 50 II
Technichal University Berlin XpaliNLP Group(柏林工业大学 XpaliNLP 组);Research Center for Artificial Intelligence (DFKI)(人工智能研究中心(DFKI));Center for European Research on Trusted AI (CERTAIN)(欧洲可信人工智能研究中心(CERTAIN))
VeriEquivBench: An Equivalence Score for Ground-Truth-Free Evaluation of Formally Verifiable Code
Huazhong University of Science and Technology(华中科技大学);University of Alberta(阿尔伯塔大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Hong Kong University of Science and Technology(香港科技大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
One Analyst Is Not Ground Truth: Grading Agent-Built Financial Models Against Observed Professional Practice
Shanghai Jiao Tong University(上海交通大学);University of Adelaide(阿德莱德大学);Tsinghua University(清华大学);National University of Singapore(新加坡国立大学);Peking University(北京大学)