本研究引入了名为Penalized Brier Score (PBS) 和 Penalized Logarithmic Loss (PLL) 的新型优越评分规则，以改进概率分类的模型评估。像 Brier Score 和 Logarithmic Loss 这样的传统评分规则，有时会为错误分类分配比正确分类更好的分数。这种与实际偏好奖励正确分类的偏差可能导致次优的模型选择。通过整合对错误分类的惩罚，PBS 和 PLL 修改了传统的严格评分规则，以始终为正确预测分配更好的分数。形式化证明表明，PBS 和 PLL 满足严格评分规则属性，同时优先奖励准确分类。实验展示了使用 PBS 和 PLL 进行模型选择、模型检查点和早停的益处。在训练期间，与 Brier Score 相比，PBS 与 F1 分数表现出更高的负相关性。因此，PBS 能更有效地识别最优检查点和早停点，从而带来更高的 F1 分数。比较分析验证了由 PBS 和 PLL 选择的模型实现了更优的 F1 分数。因此，PBS 和 PLL 通过涵盖严格评分原则和对真实分类的明确偏好，弥合了不确定性量化与准确率最大化之间的差距。所提出的指标可以增强可靠概率分类的模型评估和选择。
英文摘要：
This study introduces novel superior scoring rules called Penalized Brier Score (PBS) and Penalized Logarithmic Loss (PLL) to improve model evaluation for probabilistic classification. Traditional scoring rules like Brier Score and Logarithmic Loss sometimes assign better scores to misclassifications in comparison with correct classifications. This discrepancy from the actual preference for rewarding correct classifications can lead to suboptimal model selection. By integrating penalties for misclassifications, PBS and PLL modify traditional proper scoring rules to consistently assign better scores to correct predictions. Formal proofs demonstrate that PBS and PLL satisfy strictly proper scoring rule properties while also preferentially rewarding accurate classifications. Experiments showcase the benefits of using PBS and PLL for model selection, model checkpointing, and early stopping. PBS exhibits a higher negative correlation with the F1 score compared to the Brier Score during training. Thus, PBS more effectively identifies optimal checkpoints and early stopping points, leading to improved F1 scores. Comparative analysis verifies models selected by PBS and PLL achieve superior F1 scores. Therefore, PBS and PLL address the gap between uncertainty quantification and accuracy maximization by encapsulating both proper scoring principles and explicit preference for true classifications. The proposed metrics can enhance model evaluation and selection for reliable probabilistic classification.
Probabilistic Forecasting with Generative Networks via Scoring Rule Minimization
University of Oxford（牛津大学）；Southern University of Science and Technology（南方科技大学）；University of Warwick（华威大学）；European Centre for Medium-Range Weather Forecasts（欧洲中期天气预报中心）
Probabilistic Scoring Lists for Interpretable Machine Learning
LMU Munich（慕尼黑大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Johannes Kepler Universität Linz（约翰内斯·开普勒大学林茨分校）；Institute of Informatics（信息学研究所）；Institute for Application-oriented Knowledge Processing（应用导向知识处理研究所）
University of Munich (LMU)（慕尼黑大学（LMU））；Munich Center for Machine Learning（慕尼黑机器学习中心）；Ghent University（根特大学）
It is well known that accurate probabilistic predictors can be trained through empirical risk minimisation with proper scoring rules as loss functions. While such learners capture so-called aleatoric uncertainty of predictions, various machine learning methods have recently been developed with the goal to let the learner also represent its epistemic uncertainty, i.e., the uncertainty caused by a lack of knowledge and data. An emerging branch of the literature proposes the use of a second-order learner that provides predictions in terms of distributions on probability distributions. However, re…
Heidelberg Institute for Theoretical Studies（海德堡理论研究所）；University of Hohenheim（霍恩海姆大学）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）
A probability forecast or probabilistic classifier is reliable or calibrated if the predicted probabilities are matched by ex post observed frequencies, as examined visually in reliability diagrams. The classical binning and counting approach to plotting reliability diagrams has been hampered by a lack of stability under unavoidable, ad hoc implementation decisions. Here we introduce the CORP approach, which generates provably statistically Consistent, Optimally binned, and Reproducible reliability diagrams in an automated way. CORP is based on non-parametric isotonic regression and implemente…
Item response theory (IRT) is a non-linear generative probabilistic paradigm for using exams to identify, quantify, and compare latent traits of individuals, relative to their peers, within a population of interest. In pre-existing multidimensional IRT methods, one requires a factorization of the test items. For this task, linear exploratory factor analysis is used, making IRT a posthoc model. We propose skipping the initial factor analysis by using a sparsity-promoting horseshoe prior to perform factorization directly within the IRT model so that all training occurs in a single self-consisten…
Phonexia（Phonexia公司）；Brno University of Technology（布尔诺理工大学）；Omilia - Conversational Intelligence（Omilia对话智能公司）
In speaker recognition, where speech segments are mapped to embeddings on the unit hypersphere, two scoring backends are commonly used, namely cosine scoring or PLDA. Both have advantages and disadvantages, depending on the context. Cosine scoring follows naturally from the spherical geometry, but for PLDA the blessing is mixed -- length normalization Gaussianizes the between-speaker distribution, but violates the assumption of a speaker-independent within-speaker distribution. We propose PSDA, an analogue to PLDA that uses Von Mises-Fisher distributions on the hypersphere for both within and …
This paper focuses on multi-enrollment speaker recognition which naturally occurs in the task of online speaker clustering, and studies the properties of different scoring back-ends in this scenario. First, we show that popular cosine scoring suffers from poor score calibration with a varying number of enrollment utterances. Second, we propose a simple replacement for cosine scoring based on an extremely constrained version of probabilistic linear discriminant analysis (PLDA). The proposed model improves over the cosine scoring for multi-enrollment recognition while keeping the same performanc…
We consider the two-group classification problem and propose a kernel classifier based on the optimal scoring framework. Unlike previous approaches, we provide theoretical guarantees on the expected risk consistency of the method. We also allow for feature selection by imposing structured sparsity using weighted kernels. We propose fully-automated methods for selection of all tuning parameters, and in particular adapt kernel shrinkage ideas for ridge parameter selection. Numerical studies demonstrate the superior classification performance of the proposed approach compared to existing nonparam…
Amirkabir University of Technology (Tehran Polytechnic)（阿米尔卡比尔理工大学（德黑兰理工大学））
Credit scoring models, which are among the most potent risk management tools that banks and financial institutes rely on, have been a popular subject for research in the past few decades. Accordingly, many approaches have been developed to address the challenges in classifying loan applicants and improve and facilitate decision-making. The imbalanced nature of credit scoring datasets, as well as the heterogeneous nature of features in credit scoring datasets, pose difficulties in developing and implementing effective credit scoring models, targeting the generalization power of classification m…
Institute for Information Transmission Problems（信息传输问题研究所）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究所）
Probabilistic forecasts in the form of probability distributions over future events have become popular in several fields of statistical science. The dissimilarity between a probability forecast and an outcome is measured by a loss function (scoring rule). Popular example of scoring rule for continuous outcomes is the continuous ranked probability score (CRPS). We consider the case where several competing methods produce online predictions in the form of probability distribution functions. In this paper, the problem of combining probabilistic forecasts is considered in the prediction with expe…
Proper Scoring Rules for Right-Censored Survival Data
IDLab；Biometrics Research Group（生物信息学研究组）；Department of Electronics（电子系）；Department of Morphology, and Information Systems and Information Systems Imaging, Orthopedics（形态学系及信息系统和影像学、骨科）；Ghent University（根特大学）；Ghent University - imec（根特大学 - imec）；Rehabilitation and Nutrition（康复与营养）；Department of Electronics and Information Systems（电子系及信息系统）
Exploring Data Augmentation and Resampling Strategies for Transformer-Based Models to Address Class Imbalance in AI Scoring of Scientific Explanations in NGSS Classroom
Michigan State University（密歇根州立大学）；CREATE for STEM（STEM创新计划）；University of Houston（休斯顿大学）
RWTH Aachen University（亚琛工业大学）；Ingenieurgesellschaft Auto und Verkehr (IAV)（IAV汽车与交通工程公司）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Probabilistic models such as Gaussian processes (GPs) are powerful tools to learn unknown dynamical systems from data for subsequent use in control design. While learning-based control has the potential to yield superior performance in demanding applications, robustness to uncertainty remains an important challenge. Since Bayesian methods quantify uncertainty of the learning results, it is natural to incorporate these uncertainties into a robust design. In contrast to most state-of-the-art approaches that consider worst-case estimates, we leverage the learning method's posterior distribution i…
