在科学背景下,评估来自不确定性量化(UQ)深度学习方法的偶然不确定性估计的质量非常重要,因为不确定性具有物理意义,对其进行精确表征和解释至关重要。我们系统地比较了由两种UQ技术——深度集成(DE)和深度证据回归(DER)——测量的偶然不确定性。我们的方法聚焦于零维(0D)和二维(2D)数据,以探究UQ方法在不同数据维度下如何运作。我们研究了注入到输入和输出变量的不确定性,并包含了一种在输入不确定性情况下传播不确定性的方法,以便将预测的偶然不确定性与已知值进行比较。我们实验了三个噪声水平。所有模型和实验中预测的偶然不确定性均随注入的噪声水平而缩放。然而,对于一半的DE实验和几乎所有的DER实验,预测的不确定性在 $\ m{std}(σ_{\ m al})$ 上与真实不确定性存在校准偏差。对于2D输入不确定性实验和高噪声水平,这两种UQ方法预测的不确定性准确性最低。虽然这些结果不适用于更复杂的数据,但它们表明,对这些方法进行事后校准的进一步研究将是有益的,特别是在高噪声和高维设置中。
英文摘要:
Assessing the quality of aleatoric uncertainty estimates from uncertainty quantification (UQ) deep learning methods is important in scientific contexts, where uncertainty is physically meaningful and important to characterize and interpret exactly. We systematically compare aleatoric uncertainty measured by two UQ techniques, Deep Ensembles (DE) and Deep Evidential Regression (DER). Our method focuses on both zero-dimensional (0D) and two-dimensional (2D) data, to explore how the UQ methods function for different data dimensionalities. We investigate uncertainty injected on the input and output variables and include a method to propagate uncertainty in the case of input uncertainty so that we can compare the predicted aleatoric uncertainty to the known values. We experiment with three levels of noise. The aleatoric uncertainty predicted across all models and experiments scales with the injected noise level. However, the predicted uncertainty is miscalibrated to $\rm{std}(σ_{\rm al})$ with the true uncertainty for half of the DE experiments and almost all of the DER experiments. The predicted uncertainty is the least accurate for both UQ methods for the 2D input uncertainty experiment and the high-noise level. While these results do not apply to more complex data, they highlight that further research on post-facto calibration for these methods would be beneficial, particularly for high-noise and high-dimensional settings.
From Aleatoric to Epistemic: Exploring Uncertainty Quantification Techniques in Artificial Intelligence
University of Liverpool(利物浦大学);University of Edinburgh(爱丁堡大学);The University of Texas at Dallas(德州大学达拉斯分校);Georgia Institute of Technology(佐治亚理工学院);Emory University(埃默里大学);Simon Fraser University(西蒙菲莎大学);Kyoto University(京都大学);Zhejiang University(浙江大学);National Taiwan Normal University(国立台湾师范大学);Indiana University(印第安纳大学);Purdue University(普渡大学);AppCubic(AppCubic公司);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);The Hong Kong University of Science and Technology(香港科技大学)
Identifying Drivers of Predictive Aleatoric Uncertainty
Hasso Plattner Institute(霍普夫纳研究所);Digital Engineering Faculty(数字工程学院);University of Potsdam(波茨坦大学);Freie Universität Berlin(柏林自由大学);Department of Mathematics and Computer Science(数学与计算机科学系);Hasso Plattner Institute for Digital Health at Mount Sinai(辛克尔医学院霍普夫纳数字健康研究所);Icahn School of Medicine at Mount Sinai(辛克尔医学院)
Conformalized Generative Bayesian Imaging: An Uncertainty Quantification Framework for Computational Imaging
Department of Electrical and Computer Engineering, University of Rochester(电气与计算机工程系,罗切斯特大学);Goergen Institute for Data Science & AI, University of Rochester(数据科学与人工智能研究院,罗切斯特大学)
Quantifying Uncertainty in Discrete-Continuous and Skewed Data with Bayesian Deep Learning
Northeastern University(东北大学);risQ Inc.(risQ公司);Bay Area Environmental Research Institute(湾区环境研究所);NASA Ames Research Center(美国国家航空航天局艾姆斯研究中心);NASA Advanced Supercomputing Division(美国国家航空航天局高级超级计算部)
Cooperative Variance Estimation and Bayesian Neural Networks for Disentangling Aleatoric and Epistemic Uncertainties
Faculty of Mechanical Engineering, Delft University of Technology, Mekelweg 2, Delft, 2628 CD, The Netherlands(代尔夫特理工大学机械工程学院);School of Engineering, Brown University, 184 Hope St., Providence, RI 02912, USA(布朗大学工程学院)
Uncertainty Estimation and Out-of-Distribution Detection for LiDAR Scene Semantic Segmentation
Institute of Cartography and Geoinformatics, Leibniz University Hannover(汉诺威莱布尼茨大学制图与地理信息研究所);Institute of Photogrammetry and GeoInformation, Leibniz University Hannover(汉诺威莱布尼茨大学摄影测量与地理信息研究所)
Uncertainty Quantification in Probabilistic Machine Learning Models: Theory, Methods, and Insights
Weill Cornell Medicine, Cornell University, New York, NY(韦尔·科恩医学中心,康奈尔大学,纽约,纽约);Department of Electrical and Computer Engineering, Stony Brook University, Stony Brook, NY(电气与计算机工程系,史泰文斯布鲁克大学,史泰文斯布鲁克,纽约)
Variational LSTM with Augmented Inputs: Nonlinear Response History Metamodeling with Aleatoric and Epistemic Uncertainty
Department of Civil, Environmental, and Construction Engineering, Texas Tech University(德克萨斯理工大学土木、环境与建筑工程系);Department of Civil and Environmental Engineering, Rensselaer Polytechnic Institute(伦斯勒理工学院土木与环境工程系);National Wind Institute, Texas Tech University(德克萨斯理工大学国家风能研究所)
One step closer to unbiased aleatoric uncertainty estimation
Massachusetts Institute of Technology(麻省理工学院);Harvard University(哈佛大学);MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室);IBM Research(IBM研究院);University of California(加利福尼亚大学)
What Uncertainties Do We Need for Dynamical Systems?
Institute of Computer Science, LMU Munich(慕尼黑大学计算机科学研究所);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Department of Mathematics, LMU Munich(慕尼黑大学数学系);German Research Center for Artificial Intelligence (DFKI, DSA)(德国人工智能研究中心(DFKI, DSA))
Nonparametric Uncertainty Quantification for Single Deterministic Neural Network
Skolkovo Institute of Science and Technology(斯科尔科沃科学技术研究所);Technology Innovation Institute(技术创新研究所);HSE University(高等经济大学);Nuro, Inc.(Nuro公司);Lomonosov Moscow State University(罗蒙诺索夫莫斯科国立大学);AIRI;Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Credal and Interval Deep Evidential Classifications
Department of Computer Science, The University of Manchester(曼彻斯特大学计算机科学系);School of Engineering, Computing and Mathematics, Oxford Brookes University(奥克斯布里奇大学工程、计算与数学学院)