arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.23711cs.CL
收藏

CAT:一种基于度量的分析LLMs在受控输入变化下一致性-准确性关系的框架

CAT: A Metric-Driven Framework for Analyzing the Consistency-Accuracy Relation of LLMs under Controlled Input Variations

IBM Research Brazil(IBM巴西研究院)

Paulo Cavalin, Cassia Sanctos, Marcelo Grave, Claudio Pinhanez, Yago Primerano

AI总结:

CAT框架通过分析LLMs在可控输入变化下的准确性与一致性关系,提供了一种评估方法,结合CAR曲线和CORE指数,量化准确率与一致性的权衡。

摘要:

我们介绍了CAT,一种框架,旨在评估和可视化在可控输入变化下大型语言模型(LLMs)的准确性和响应一致性之间的相互作用,使用多项选择(MC)基准作为案例研究。当前的评估实践主要关注模型能力,如准确率或基准分数,更近期的测量一致性被视为部署LLMs在高风险、现实应用中的关键属性。本文认为,尽管两个维度仍应独立评估,但它们的相互依赖性也需要考虑,以更细致地评估LLMs。CAT的核心是“一致性-准确性关系(CAR)”曲线,它可视化了随着一致性要求增加,模型准确率如何变化,如通过“最小一致性准确率(MCA)”指标定义。我们进一步提出了“一致性导向的鲁棒性估计(CORE)”指数,这是一个全局指标,结合CAR曲线的面积和形状,以量化准确率和一致性之间的权衡。我们展示了该框架在多种通用和领域特定LLMs上进行多个MC基准测试的实践演示。我们还概述了如何将CAT扩展到MC任务之外,通过可调整的评分函数支持长形式、开放性评估。

英文摘要:

We introduce \textsc{CAT}, a framework designed to evaluate and visualize the \emph{interplay} of \emph{accuracy} and \emph{response consistency} of Large Language Models (LLMs) under controllable input variations, using multiple-choice (MC) benchmarks as a case study. Current evaluation practices primarily focus on model capabilities such as accuracy or benchmark scores and, more recently, measuring consistency is being considered an essential property for deploying LLMs in high-stake, real-world applications. We argue in this paper that although both dimensions should still be evaluated independently, their inter-dependency also need to be considered for a more nuanced evaluation of LLMs. At the core of \textsc{CAT} are the \emph{Consistency-Accuracy Relation (CAR)} curves, which visualize how model accuracy varies with increasing consistency requirements, as defined by the \emph{Minimum-Consistency Accuracy (MCA)} metric. We further propose the \emph{Consistency-Oriented Robustness Estimate (CORE)} index, a global metric that combines the area and shape of the CAR curve to quantify the trade-off between accuracy and consistency. We present a practical demonstration of our framework across a diverse set of generalist and domain-specific LLMs, evaluated on multiple MC benchmarks. We also outline how \textsc{CAT} can be extended beyond MC tasks to support long-form, open-ended evaluations through adaptable scoring functions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑