Estimating how well a machine learning model performs during inference is critical in a variety of scenarios (for example, to quantify uncertainty, or to choose from a library of available models). However, the standard accuracy estimate of softmax confidence is not versatile and cannot reliably predict different performance metrics (e.g., F1-score, recall) or the performance in different application scenarios or input domains. In this work, we systematically generalize performance estimation to a diverse set of metrics and scenarios and discuss generalized notions of uncertainty calibration. We propose the use of post-hoc models to accomplish this goal and investigate design parameters, including the model type, feature engineering, and performance metric, to achieve the best estimation quality. Emphasis is given to object detection problems and, unlike prior work, our approach enables the estimation of per-image metrics such as recall and F1-score. Through extensive experiments with computer vision models and datasets in three use cases -- mobile edge offloading, model selection, and dataset shift -- we find that proposed post-hoc models consistently outperform the standard calibrated confidence baselines. To the best of our knowledge, this is the first work to develop a unified framework to address different performance estimation problems for machine learning inference.
Rethinking Post-Hoc Calibration in Semantic Segmentation
ICube Laboratory, CNRS UMR 7357, University of Strasbourg（斯特拉斯堡大学ICube实验室，CNRS UMR 7357）；CLCC Institut Strauss（斯特劳斯研究所）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；Faculty of Mathematics and Computer Science, University of Heidelberg（海德堡大学数学与计算机科学学院）；Medical Faculty Heidelberg, Heidelberg University（海德堡大学医学院）；Pattern Analysis and Learning Group, Dept. of Radiation Oncology, Heidelberg University Hospital（海德堡大学医院放射肿瘤科模式分析与学习组）
Train Once, Forget Precisely: Anchored Optimization for Efficient Post-Hoc Unlearning
University College London, London, United Kingdom（伦敦大学学院）；University of Nottingham, Nottingham, United Kingdom（诺丁汉大学）；Bharati Vidyapeeth College of Engineering, New Delhi, India（巴里蒂大学工程学院）
Technical University of Munich（慕尼黑工业大学）；Helmholtz Munich（亥姆霍兹慕尼黑中心）；ELLIS Institute Finland（芬兰ELLIS研究所）；Aalto University（阿尔托大学）；Finnish Center for Artificial Intelligence（芬兰人工智能中心）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Munich Data Science Institute（慕尼黑数据科学研究所）；Ericsson Research（爱立信研究院）；KTH Royal Institute of Technology（KTH皇家理工学院）；Digital Futures（数字未来研究所）
Post-hoc Overall Survival Time Prediction from Brain MRI
Australian Institute for Machine Learning, The University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；Institute for Systems and Robotics, Instituto Superior Tecnico（高等理工学院系统与机器人研究所）
Evaluating Post-hoc Interpretability with Intrinsic Interpretability
University of Coimbra（科英布拉大学）；IPO-Porto Research Centre（波尔图肿瘤学研究所研究中心）；University of Applied Sciences Western Switzerland (HES-SO Valais)（瑞士西部应用科学大学（瓦莱州高等专业学院））
Deep Attention Reweighting: Post-Hoc Attention-Based Feature Aggregation in CNNs for Disentangling Core and Spurious Features under Spurious Correlations
Improved post-hoc probability calibration for out-of-domain MRI segmentation
Imperial College London（帝国理工学院）；Fudan University（复旦大学）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Technical University of Munich（慕尼黑工业大学）
On the Faithfulness of Post-Hoc Concept Bottleneck Models
Institute of Data Science, German Aerospace Center（数据科学研究所，德国航空航天中心）；Computer Vision Group Jena, Friedrich Schiller University Jena（贾恩计算机视觉小组，弗里德里希·席勒大学贾恩）；GEOMAR Helmholtz Centre for Ocean Research Kiel（基尔海洋研究赫尔姆霍茨中心）
Single-weight Model Editing for Post-hoc Spurious Correlation Neutralization
Department of Computer Science and Software Engineering, University of Western Australia（西澳大学计算机科学与软件工程系）；School of Computing and Information Systems, University of Melbourne（墨尔本大学计算机科学与信息系）
