温度缩放分类器:温度缩放的一些基本性质
The Well-Tempered Classifier: Some Elementary Properties of Temperature Scaling
AI总结:
本文通过信息投影和线性缩放子模型等新视角,严格分析了温度缩放对分类器校准和LLM多样性的影响,证明升温普遍增加不确定性但质疑其增加多样性的说法。
摘要:
温度缩放是一种简单的方法,可以控制概率模型的不确定性。它主要用于两个场景:改进分类器的校准和调节大型语言模型(LLM)的随机性。在这两种情况下,温度缩放都是最流行的方法。尽管其流行,但温度缩放性质的严格理论分析仍然难以捉摸。我们在此研究其中一些性质。对于分类,我们表明提高温度在非常普遍的意义上增加了模型的不确定性(特别是增加了其熵)。然而,对于LLM,我们质疑了提高温度会增加多样性的常见说法。此外,我们引入了温度缩放的两种新表征。第一种是几何的:温度缩放模型被证明是原始模型在具有给定熵的模型集合上的信息投影。第二种表征阐明了温度缩放作为更一般线性缩放器(如矩阵缩放和狄利克雷校准)的子模型的作用:我们表明温度缩放是唯一不改变模型硬预测的线性缩放器。
英文摘要:
Temperature scaling is a simple method that allows to control the uncertainty of probabilistic models. It is mostly used in two contexts: improving the calibration of classifiers and tuning the stochasticity of large language models (LLMs). In both cases, temperature scaling is the most popular method for the job. Despite its popularity, a rigorous theoretical analysis of the properties of temperature scaling has remained elusive. We investigate here some of these properties. For classification, we show that increasing the temperature increases the uncertainty in the model in a very general sense (and in particular increases its entropy). However, for LLMs, we challenge the common claim that increasing temperature increases diversity. Furthermore, we introduce two new characterisations of temperature scaling. The first one is geometric: the tempered model is shown to be the information projection of the original model onto the set of models with a given entropy. The second characterisation clarifies the role of temperature scaling as a submodel of more general linear scalers such as matrix scaling and Dirichlet calibration: we show that temperature scaling is the only linear scaler that does not change the hard predictions of the model.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。