Evaluating the Correctness of Inference Patterns Used by LLMs for Judgment
Shanghai Jiao Tong University（上海交通大学）；The Chinese University of Hong Kong（香港中文大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）；Beijing Institute for General Artificial Intelligence（北京一般人工智能研究所）
Lu Chen, Yuxuan Huang, Yixing Li, Dongrui Liu, Qihan Ren, Shuai Zhao, Kun Kuang, Zilong Zheng, Quanshi Zhang
This paper presents a method to analyze the inference patterns used by Large Language Models (LLMs) for judgment in a case study on legal LLMs, so as to identify potential incorrect representations of the LLM, according to human domain knowledge. Unlike traditional evaluations on language generation results, we propose to evaluate the correctness of the detailed inference patterns of an LLM behind its seemingly correct outputs. To this end, we quantify the interactions between input phrases used by the LLM as primitive inference patterns, because recent theoretical achievements have proven several mathematical guarantees of the faithfulness of the interaction-based explanation. We design a set of metrics to evaluate the detailed inference patterns of LLMs. Experiments show that even when the language generation results appear correct, a significant portion of the inference patterns used by the LLM for the legal judgment may represent misleading or irrelevant logic.
SiMing-Bench: Evaluating Procedural Correctness from Continuous Interactions in Clinical Skill Videos
School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；Center for Language and Information Research, Wuhan University（武汉大学语言与信息研究中心）；Southwest Jiaotong University（西南交通大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；The University of Manchester（曼彻斯特大学）；Zhongnan Hospital of Wuhan University（武汉大学中南医院）
ClinDet-Bench: Beyond Abstention, Evaluating Judgment Determinability of LLMs in Clinical Decision-Making
Kyoto University, Department of Biomedical Data Intelligence（京都大学生物医学数据智能系）；Kyoto University, Department of Cardiovascular Medicine（京都大学心血管医学系）；The University of Tokyo（东京大学）
Fact or Guesswork? Evaluating Large Language Models' Medical Knowledge with Structured One-Hop Judgments
University of Georgia（佐治亚大学）；University of California, Merced（加州大学默塞德分校）；Lehigh University（莱特大学）；The University of Queensland（昆士兰大学）；National University of Singapore（新加坡国立大学）；University of California, Los Angeles（加州大学洛杉矶分校）
Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English
HiTZ Center, University of the Basque Country (UPV/EHU)（巴斯克大学（UPV/EHU）HiTZ中心）；University of Alicante (UA)（阿利坎特大学（UA））；GOI Institute, Basque Summer University (UEU)（巴斯克夏季大学（UEU）GOI研究所）
Who Owns That? Evaluating Ownership Intuitions in Large Language Models
College of AI, Tsinghua University（清华大学人工智能学院）；Department of Psychological and Cognitive Sciences, Tsinghua University（清华大学心理与认知科学系）；Faculty of Psychology, Beijing Normal University（北京师范大学心理学部）
The illusion of a perfect metric: Why evaluating AI's words is harder than it looks
Iris AI Borgen, Norway（Iris AI 布尔根，挪威）；Iris AI Neurobiology BAS Sofia, Bulgaria（Iris AI 神经生物学 BAS 萨福ia，保加利亚）；Iris AI Sofia, Bulgaria（Iris AI 萨福ia，保加利亚）
How Long Reasoning Chains Influence LLMs' Judgment of Answer Factuality
State Key Laboratory of AI Safety（人工智能安全国家重点实验室）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing University of Post and Telecommunications（北京邮电大学）
ORCA: Open-ended Response Correctness Assessment for Audio Question Answering
Brno University of Technology（布尔诺科技大学）；Universidad Autónoma de Madrid（马德里自治大学）；University of Buenos Aires（布宜诺斯艾利斯大学）；Tufts University（塔夫茨大学）；University of Maryland（马里兰大学）
CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environments
School of Mathematics and Sciences, Fudan University（复旦大学数学科学学院）；Shanghai Center for Mathematical Sciences, Fudan University（复旦大学上海数学中心）；Alibaba Group（阿里巴巴集团）；Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University（复旦大学类脑智能科学与技术研究院）；Center for Applied Mathematics & Shanghai Key Laboratory of Contemporary Applied Mathematics, Fudan University（复旦大学应用数学中心暨上海市现代应用数学重点实验室）
