How Certain are Uncertainty Estimates? Three Novel Earth Observation Datasets for Benchmarking Uncertainty Quantification in Machine Learning
Technical University of Munich（慕尼黑工业大学）；National University of Sciences & Technology Pakistan（巴基斯坦国立科技大学）；German Aerospace Center（德国航空航天中心）；University of Bristol（布里斯托大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）
Yuanyuan Wang, Qian Song, Dawood Wasif, Muhammad Shahzad, Christoph Koller, Jonathan Bamber, Xiao Xiang Zhu
备注：
Submitted to IEEE Geoscience and Remote Sensing Magazine
Uncertainty quantification (UQ) is essential for assessing the reliability of Earth observation (EO) products. However, the extensive use of machine learning models in EO introduces an additional layer of complexity, as those models themselves are inherently uncertain. While various UQ methods do exist for machine learning models, their performance on EO datasets remains largely unevaluated. A key challenge in the community is the absence of the ground truth for uncertainty, i.e. how certain the uncertainty estimates are, apart from the labels for the image/signal. This article fills this gap by introducing three benchmark datasets specifically designed for UQ in EO machine learning models. These datasets address three common problem types in EO: regression, image segmentation, and scene classification. They enable a transparent comparison of different UQ methods for EO machine learning models. We describe the creation and characteristics of each dataset, including data sources, preprocessing steps, and label generation, with a particular focus on calculating the reference uncertainty. We also showcase baseline performance of several machine learning models on each dataset, highlighting the utility of these benchmarks for model development and comparison. Overall, this article offers a valuable resource for researchers and practitioners working in artificial intelligence for EO, promoting a more accurate and reliable quality measure of the outputs of machine learning models. The dataset and code are accessible via https://gitlab.lrz.de/ai4eo/WG_Uncertainty.
Uncertainty quantification for probabilistic machine learning in earth observation using conformal prediction
Stellenbosch University（斯泰伦博斯大学）；The Nature Conservancy（大自然保护协会）；University of Cape Town（开普敦大学）；Norwegian Institute for Nature Research—NINA（挪威自然研究所—NINA）；Jet Propulsion Laboratory（喷气推进实验室）；California Institute of Technology（加州理工学院）；South African Environmental Observation Network（南非环境观测网络）
Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph
MBZUAI（穆罕默德·本·扎耶德人工智能大学）；ETH Zürich（苏黎世联邦理工学院）；Skoltech（斯科尔科沃科学技术学院）；AIRI（人工智能研究所）；HSE University（高等经济学院）；FRC CSC RAS（俄罗斯科学院计算机科学控制问题科学中心）；Nebius（Nebius公司）；The University of Melbourne（墨尔本大学）
Decision-Oriented Uncertainty Quantification for Risk Control in Earth System Spatiotemporal Foundation Models
Vanderbilt University（范德堡大学）；City University of New York（纽约城市大学）；Yale University（耶鲁大学）；Wyze Inc.（Wyze公司）；Northeastern University（东北大学）；University of Southern California（南加州大学）；Southwest University（西南大学）
Beyond Point Estimates: Benchmarking Uncertainty Quantification Methods on the AION-1 Astronomical Foundation Model
Scientific Computing Division Fermi National Accelerator Laboratory（费米国家加速器实验室科学计算部）；Fermi National Accelerator Laboratory（费米国家加速器实验室）；Department of Astronomy and Astrophysics University of Chicago（芝加哥大学天文学与天体物理学系）；NSF and Simons SkAI Institute（国家科学基金会与Simons SkAI研究所）；Google DeepMind（谷歌DeepMind）
Evidential Deep Learning: Enhancing Predictive Uncertainty Estimation for Earth System Science Applications
National Center for Atmospheric Research（国家大气研究中心）；Cooperative Institute for Severe and High-Impact Weather Research and Operations（恶劣与高影响天气研究与合作研究所）；National Severe Storms Laboratory（国家强风暴实验室）；University Corporation for Atmospheric Research（大学大气研究公司）；University of Maryland（马里兰大学）；University at Albany, State University of New York（纽约州立大学奥尔巴尼分校）；Colorado State University（科罗拉多州立大学）；The Pennsylvania State University（宾夕法尼亚州立大学）
Towards Efficient Benchmarking of Foundation Models in Remote Sensing: A Capabilities Encoding Approach
IRISA, Université Bretagne Sud, UMR 6074（IRISA，布列塔尼大学，UMR 6074）；Centre National d’Études Spatiales (CNES)（国家空间科学研究中心（CNES））；UiT The Arctic University of Norway（挪威北极大学）
Evidential Deep Learning for Uncertainty Quantification and Out-of-Distribution Detection in Jet Identification using Deep Neural Networks
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；The Grainger College of Engineering（格兰杰工程学院）；Center for Artificial Intelligence Innovation（人工智能创新中心）；National Center for Supercomputing Applications（国家超级计算应用中心）；Siebel School of Computing and Data Science（西贝尔计算与数据科学学院）
Probabilistic Machine Learning for Noisy Labels in Earth Observation
Orion Lab（猎户座实验室）；National Observatory of Athens（雅典国家天文台）；National Technical University of Athens（雅典国立技术大学）；Image Processing Laboratory (IPL), Universitat de València（瓦伦西亚大学图像处理实验室）；Harokopio University of Athens（哈罗科皮奥大学）；Archimedes, Athena Research Center（雅典娜研究中心阿基米德分部）
Reconstruction of Solar EUV Irradiance Using CaII K Images and SOHO/SEM Data with Bayesian Deep Learning and Uncertainty Quantification
Institute for Space Weather Sciences, New Jersey Institute of Technology, University Heights, Newark, NJ 07102, USA（空间天气科学研究所，新泽西理工学院，大学 Heights，新泽西州诺克斯维尔，NJ 07102，美国）；Department of Computer Science, Sam Houston State University, Huntsville, TX 77341, USA（计算机科学系，萨姆休斯顿州立大学，休斯顿，德克萨斯州77341，美国）；Center for Solar-Terrestrial Research, New Jersey Institute of Technology, University Heights, Newark, NJ 07102, USA（太阳-地球研究中心，新泽西理工学院，大学 Heights，新泽西州诺克斯维尔，NJ 07102，美国）；Department of Computer Science, New Jersey Institute of Technology, University Heights, Newark, NJ 07102, USA（计算机科学系，新泽西理工学院，大学 Heights，新泽西州诺克斯维尔，NJ 07102，美国）；Big Bear Solar Observatory, New Jersey Institute of Technology, 40386 North Shore Lane, Big Bear City, CA 92314, USA（大熊太阳观测站，新泽西理工学院，40386 North Shore Lane，大熊城，加利福尼亚州92314，美国）
Benchmarking Uncertainty and its Disentanglement in multi-label Chest X-Ray Classification
Fraunhofer Heinrich-Hertz-Institut（弗劳恩霍夫海因里希-赫兹研究所）；Technische Universität Berlin（柏林技术大学）；The Berlin Institute for the Foundations of Learning and Data (BIFOLD)（柏林学习与数据基础研究所）
GeoChrono: Benchmarking and Rethinking Long-Term Temporal Understanding in Remote Sensing
Beijing University of Posts and Telecommunications（北京邮电大学）；Tsinghua University（清华大学）；Hunan Normal University（湖南师范大学）；City University of Hong Kong（香港城市大学）
A Perspective on Gaussian Processes for Earth Observation
Universitat de València（瓦伦西亚大学）；University of Oxford（牛津大学）；German Aerospace Center（德国航空航天中心）；Max Planck Institute for Biogeochemistry（马克斯·普朗克生物地球化学研究所）
Regularization of ML models for Earth systems by using longer model timesteps
University of Cambridge（剑桥大学）；British Antarctic Survey（英国南极调查局）；Helmholtz Centre for Environmental Research – UFZ（亥姆霍兹环境研究中心（UFZ））；University of Oxford（牛津大学）；European Centre for Medium-Range Weather Forecasts（欧洲中期天气预报中心）；Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI)（可扩展数据分析与人工智能中心（ScaDS.AI））
Benchmarking Diffusion Annealing-Based Bayesian Inverse Problem Solvers
Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin（奥登计算工程与科学研究所，德克萨斯大学奥斯汀分校）；Dept of Biomedical Engineering, The University of Texas at Austin（生物医学工程系，德克萨斯大学奥斯汀分校）
Open-Insect: Benchmarking Open-Set Recognition of Novel Species in Biodiversity Monitoring
McGill University（麦吉尔大学）；Mila - Quebec Artificial Intelligence Institute（魁北克人工智能研究所）；University of Copenhagen（哥本哈根大学）；Agriculture and Agri-food Canada（加拿大农业与农业食品部）；Smithsonian Tropical Research Institute（史密松尼热带研究所）；Biology Center, Czech Academy of Sciences（捷克科学院生物学中心）；Maestria de Entomologia, University of Panama（巴拿马大学昆虫学系）；Massachusetts Institute of Technology（麻省理工学院）；Montréal Insectarium（蒙特利尔昆虫馆）
