首页/论文/2001.11242

arXiv 2001.112422020-01-30cs.LGstat.ML

Better Multi-class Probability Estimates for Small Data Sets

University of Oulu（奥卢大学）；Oura Health Ltd.（欧拉开健康有限公司）

Tuomo Alasalmi, Jaakko Suutala, Heli Koskimäki, Juha Röning

英文摘要：

Many classification applications require accurate probability estimates in addition to good class separation but often classifiers are designed focusing only on the latter. Calibration is the process of improving probability estimates by post-processing but commonly used calibration algorithms work poorly on small data sets and assume the classification task to be binary. Both of these restrictions limit their real-world applicability. Previously introduced Data Generation and Grouping algorithm alleviates the problem posed by small data sets and in this article, we will demonstrate that its application to multi-class problems is also possible which solves the other limitation. Our experiments show that calibration error can be decreased using the proposed approach and the additional computational cost is acceptable.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Information-Geometric Set Embeddings (IGSE): From Sets to Probability Distributions

CSIRO Data61（澳大利亚联邦科学与工业研究组织数据61研究所）；Sony Computer Science Laboratories, Inc.（索尼计算机科学实验室公司）

This letter introduces an abstract learning problem called the "set embedding": The objective is to map sets into probability distributions so as to lose less information. We relate set union and intersection operations with corresponding interpolations of probability distributions. We also demonstrate a preliminary solution with experimental results on toy set embedding examples.

2019-12-11cs.LGstat.ML

Bagged $k$-Distance for Mode-Based Clustering Using the Probability of Localized Level Sets

University of Twente（特文特大学）

In this paper, we propose an ensemble learning algorithm named \textit{bagged $k$-distance for mode-based clustering} (\textit{BDMBC}) by putting forward a new measurement called the \textit{probability of localized level sets} (\textit{PLLS}), which enables us to find all clusters for varying densities with a global threshold. On the theoretical side, we show that with a properly chosen number of nearest neighbors $k_D$ in the bagged $k$-distance, the sub-sample size $s$, the bagging rounds $B$, and the number of nearest neighbors $k_L$ for the localized level sets, BDMBC can achieve optimal …

2022-10-18cs.LGstat.ML

Learning Confidence Sets using Support Vector Machines

Binghamton University, State University of New York（纽约州立大学宾汉姆顿大学）

The goal of confidence-set learning in the binary classification setting is to construct two sets, each with a specific probability guarantee to cover a class. An observation outside the overlap of the two sets is deemed to be from one of the two classes, while the overlap is an ambiguity region which could belong to either class. Instead of plug-in approaches, we propose a support vector classifier to construct confidence sets in a flexible manner. Theoretically, we show that the proposed learner can control the non-coverage rates and minimize the ambiguity with high probability. Efficient al…

2018-09-28cs.LGstat.ML

Better Classifier Calibration for Small Data Sets

University of Oulu（奥卢大学）；Oura Health Ltd.（欧拉健康有限公司）

Classifier calibration does not always go hand in hand with the classifier's ability to separate the classes. There are applications where good classifier calibration, i.e. the ability to produce accurate probability estimates, is more important than class separation. When the amount of data for training is limited, the traditional approach to improve calibration starts to crumble. In this article we show how generating more data for calibration is able to improve calibration algorithm performance in many cases where a classifier is not naturally producing well-calibrated outputs and the tradi…

2020-05-25cs.LGstat.ML

Probability density estimation for sets of large graphs with respect to spectral information using stochastic block models

University of Colorado at Boulder（科罗拉多大学博尔德分校）

For graph-valued data sampled iid from a distribution $μ$, the sample moments are computed with respect to a choice of metric. In this work, we equip the set of graphs with the pseudo-metric defined by the $\ell_2$ norm between the eigenvalues of the respective adjacency matrices. We use this pseudo metric and the respective sample moments of a graph valued data set to infer the parameters of a distribution $\hatμ$ and interpret this distribution as an approximation of $μ$. We verify experimentally that complex distributions $μ$ can be approximated well taking this approach.

2022-07-05cs.LG

Nonparametric Estimation of Uncertainty Sets for Robust Optimization

We investigate a data-driven approach to constructing uncertainty sets for robust optimization problems, where the uncertain problem parameters are modeled as random variables whose joint probability distribution is not known. Relying only on independent samples drawn from this distribution, we provide a nonparametric method to estimate uncertainty sets whose probability mass is guaranteed to approximate a given target mass within a given tolerance with high confidence. The nonparametric estimators that we consider are also shown to obey distribution-free finite-sample performance bounds that …

2020-09-20cs.LGmath.OC

更多相关论文收起更多论文

On Classification with Bags, Groups and Sets

Many classification problems can be difficult to formulate directly in terms of the traditional supervised setting, where both training and test samples are individual feature vectors. There are cases in which samples are better described by sets of feature vectors, that labels are only available for sets rather than individual samples, or, if individual labels are available, that these are not independent. To better deal with such problems, several extensions of supervised learning have been proposed, where either training and/or test objects are sets of feature vectors. However, having been …

2014-10-07cs.CVcs.LGstat.ML

基于标签排序的深度分类器共形预测

Conformal Prediction for Deep Classifier via Label Ranking

Southern University of Science and Technology（南方科技大学）；ShanghaiTech University（上海科技大学）；Rutgers University（罗格斯大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Chongqing University（重庆大学）

针对深度分类器概率校准不佳导致共形预测集过大的问题，本文提出 SAPS 算法，通过仅保留最大 softmax 概率来降低非一致性分数对概率值的依赖，从而生成更紧凑的预测集并提升条件覆盖率。

2024-06-06cs.CVcs.LGmath.ST

Superposition as Data Augmentation using LSTM and HMM in Small Training Sets

Novosibirsk State University（新西伯利亚国立大学）

Considering audio and image data as having quantum nature (data are represented by density matrices), we achieved better results on training architectures such as 3-layer stacked LSTM and HMM by mixing training samples using superposition augmentation and compared with plain default training and mix-up augmentation. This augmentation technique originates from the mix-up approach but provides more solid theoretical reasoning based on quantum properties. We achieved 3% improvement (from 68% to 71%) by using 38% lesser number of training samples in Russian audio-digits recognition task and 7,16% …

2019-10-24cs.LGeess.AS

Is the Best Better? Bayesian Statistical Model Comparison for Natural Language Processing

Wrocław University of Science and Technology（弗罗茨瓦夫科技大学）；City University of New York（纽约城市大学）

Recent work raises concerns about the use of standard splits to compare natural language processing models. We propose a Bayesian statistical model comparison technique which uses k-fold cross-validation across multiple data sets to estimate the likelihood that one model will outperform the other, or that the two will produce practically equivalent results. We use this technique to rank six English part-of-speech taggers across two data sets and three evaluation metrics.

2020-10-06cs.CLcs.LGstat.ME