We introduce \textsc{CAT}, a framework designed to evaluate and visualize the \emph{interplay} of \emph{accuracy} and \emph{response consistency} of Large Language Models (LLMs) under controllable input variations, using multiple-choice (MC) benchmarks as a case study. Current evaluation practices primarily focus on model capabilities such as accuracy or benchmark scores and, more recently, measuring consistency is being considered an essential property for deploying LLMs in high-stake, real-world applications. We argue in this paper that although both dimensions should still be evaluated independently, their inter-dependency also need to be considered for a more nuanced evaluation of LLMs. At the core of \textsc{CAT} are the \emph{Consistency-Accuracy Relation (CAR)} curves, which visualize how model accuracy varies with increasing consistency requirements, as defined by the \emph{Minimum-Consistency Accuracy (MCA)} metric. We further propose the \emph{Consistency-Oriented Robustness Estimate (CORE)} index, a global metric that combines the area and shape of the CAR curve to quantify the trade-off between accuracy and consistency. We present a practical demonstration of our framework across a diverse set of generalist and domain-specific LLMs, evaluated on multiple MC benchmarks. We also outline how \textsc{CAT} can be extended beyond MC tasks to support long-form, open-ended evaluations through adaptable scoring functions.
Are Economists Always More Introverted? Analyzing Consistency in Persona-Assigned LLMs
Research Centre for Information Systems Engineering (LIRIS), KU Leuven(信息系统工程研究中心(LIRIS),鲁汶大学);Department of Engineering Management, University of Antwerp(工程管理系,安特卫普大学);Department of Decision Analytics and Risk, University of Southampton(决策分析与风险系,南安普顿大学);School of Information, University of Michigan(信息学院,密歇根大学)
To Know or Not To Know? Analyzing Self-Consistency of Large Language Models under Ambiguity
LMU Munich(慕尼黑大学);Faculty of Philological and Cultural Studies, University of Vienna(维也纳大学语言与文化学院);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);UniVie Doctoral School Computer Science(维也纳大学计算机博士学院)
Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);School of Intelligence Science and Technology, University of Science and Technology Beijing(北京科技大学智能科学与技术学院)
Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuning
College of Information Sciences and Technology, Pennsylvania State University(宾夕法尼亚州立大学信息科学与技术学院);School of Computer Science and Technology, Singapore Management University(新加坡管理大学计算机科学与技术学院);School of Computer Science, Hefei Univeristy of Technology(合肥工业大学计算机学院)
On Analyzing Annotation Consistency in Online Abusive Behavior Datasets
University of Saskatchewan(萨斯喀彻温大学);University of Electronic Science and Technology of China(电子科技大学);Istituto Dalle Molle di Studi sull’Intelligenza Artificiale(达莱莫勒人工智能研究所)
Proposing Topic Models and Evaluation Frameworks for Analyzing Associations with External Outcomes: An Application to Leadership Analysis Using Large-Scale Corporate Review Data
Accenture Japan Ltd(安永日本有限公司);Kyoto University Institute for the Future of Human Society(京都大学未来人类社会研究所);OpenWork Inc.(OpenWork公司);Integral Corporation(Integral公司)