Pre-trained Transformers are now ubiquitous in natural language processing, but despite their high end-task performance, little is known empirically about whether they are calibrated. Specifically, do these models' posterior probabilities provide an accurate empirical measure of how likely the model is to be correct on a given example? We focus on BERT and RoBERTa in this work, and analyze their calibration across three tasks: natural language inference, paraphrase detection, and commonsense reasoning. For each task, we consider in-domain as well as challenging out-of-domain settings, where models face more examples they should be uncertain about. We show that: (1) when used out-of-the-box, pre-trained models are calibrated in-domain, and compared to baselines, their calibration error out-of-domain can be as much as 3.5x lower; (2) temperature scaling is effective at further reducing calibration error in-domain, and using label smoothing to deliberately increase empirical uncertainty helps calibrate posteriors out-of-domain.
Technical University of Denmark（丹麦技术大学）；University of Oxford（牛津大学）；Danish Research Centre for Magnetic Resonance（丹麦磁共振研究中心）；Copenhagen University Hospital Hvidovre（哥本哈根大学哈维德夫医院）
Imputation-free transformer learning enables robust Alzheimer's disease prediction and calibrated uncertainty quantification across heterogeneous clinical cohorts
Platform of Bioinformatics（生物信息平台）；Lausanne University Hospital（洛桑大学医院）；Faculty of Biology and Medicine（生物学与医学学院）；University of Lausanne（洛桑大学）；Leenaards Memory Centre（Leenaards记忆中心）；Department of Clinical Neurosciences（临床神经科学系）
Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shift
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Bosch Center for Artificial Intelligence（博世人工智能中心）；FZI Research Center for Information Technology（FZI信息技术研究中心）
Bayesian Transformer for Probabilistic Load Forecasting in Smart Grids
O&M Analytics, AES Clean Energ（O&M分析部，AES清洁能源）；The AES Corporation（AES公司）；Department of Information and Communication Engineering（信息与通信工程系）；Pabna University of Science and Technology（帕纳大学科学与技术）
Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformers
U2IS, ENSTA, Institut Polytechnique de Paris（U2IS、ENSTA、巴黎理工学院）；University of Trento（特伦托大学）；NVIDIA；University of Bergamo, Italy（意大利贝拉姆博大学）；Fondazione Bruno Kessler (FBK)（布鲁诺·凯斯勒基金会）；AMIAD, Pôle Recherche, Palaiseau（AMIAD、研究部、帕莱索）
Do Schwartz Higher-Order Values Help Sentence-Level Human Value Detection? A Study of Hierarchical Gating and Calibration
PRHLT Research Center, Universitat Politècnica de València（PRHLT研究中心，瓦伦西亚理工大学）；School of Science, Engineering and Design, Universidad Europea de Valencia（瓦伦西亚欧洲大学科学、工程与设计学院）；Valencian Graduate School and Research Network of Artificial Intelligence (ValgrAI)（瓦伦西亚人工智能研究生院与研究网络（ValgrAI））
Human Values in a Single Sentence: Moral Presence, Hierarchies, and Transformer Ensembles on the Schwartz Continuum
PRHLT Research Center, Universitat Politècnica de València（PRHLT研究中心，巴塞罗那理工大学）；School of Science, Engineering and Design, Universidad Europea de Valencia（科学、工程与设计学院，欧洲大学瓦伦西亚分校）；Valencian Graduate School（瓦伦西亚研究生学院）；Research Network of Artificial Intelligence (ValgrAI)（人工智能研究网络（ValgrAI））
