The success of deep learning (DL) fostered the creation of unifying frameworks such as tensorflow or pytorch as much as it was driven by their creation in return. Having common building blocks facilitates the exchange of, e.g., models or concepts and makes developments easier replicable. Nonetheless, robust and reliable evaluation and assessment of DL models has often proven challenging. This is at odds with their increasing safety relevance, which recently culminated in the field of "trustworthy ML". We believe that, among others, further unification of evaluation and safeguarding methodologies in terms of toolkits, i.e., small and specialized framework derivatives, might positively impact problems of trustworthiness as well as reproducibility. To this end, we present the first survey on toolkits for uncertainty estimation (UE) in DL, as UE forms a cornerstone in assessing model reliability. We investigate 11 toolkits with respect to modeling and evaluation capabilities, providing an in-depth comparison for the three most promising ones, namely Pyro, Tensorflow Probability, and Uncertainty Quantification 360. While the first two provide a large degree of flexibility and seamless integration into their respective framework, the last one has the larger methodological scope.
Incorporating uncertainty quantification into travel mode choice modeling: a Bayesian neural network (BNN) approach and an uncertainty-guided active survey framework
Bridging AI and Energy Forecasting: An Autonomous Workflow with Customized Toolkit
The University of Hong Kong(香港大学);Shenzhen Institute of Research and Innovation, The University of Hong Kong (Shenzhen)(香港大学(深圳)深圳研究院);Zhejiang Normal University(浙江师范大学);DAMO Academy, Alibaba Group (U.S.) Inc.(阿里巴巴集团达摩院(美国)公司);Griffith University(格里菲斯大学)
PyHealth 2.0: A Comprehensive Open-Source Toolkit for Accessible and Reproducible Clinical Deep Learning
University of Illinois Urbana-Champaign, Urbana, IL, USA(伊利诺伊大学厄巴纳-香槟分校);PyHealth Research Initiative(PyHealth研究计划);University of Illinois College of Medicine, Chicago, IL, USA(伊利诺伊大学医学院);The University of Edinburgh, Edinburgh, UK(爱丁堡大学);Health Data Research UK, London, UK(英国健康数据研究);Department of Biomedical Engineering, Bangladesh University of Engineering(孟加拉国工程大学生物医学工程系)
Unveil Sources of Uncertainty: Feature Contribution to Conformal Prediction Intervals
Département de Mathématiques, Université du Québec à Montréal(魁北克大学蒙特利尔分校数学系);Institut Intelligence et Données, Université Laval(拉瓦尔大学智能与数据研究所);CNRS - Université de Montréal(法国国家科学研究中心-蒙特利尔大学);CRM – CNRS Aix Marseille Univ, CNRS, AMSE, Marseille, France(法国国家科学研究中心-阿维尼昂大学)
How Many Human Survey Respondents is a Large Language Model Worth? An Uncertainty Quantification Perspective
Department of IEOR, Columbia University(哥伦比亚大学工业工程与运筹学系);Decision, Risk, and Operations Division, Columbia Business School(哥伦比亚商学院决策、风险与运营分校);Department of IEOR and Data Science Institute, Columbia University(哥伦比亚大学工业工程与运筹学系及数据科学研究所)
Interpretable Human-Label-Free Deep Learning for Real-Bogus Classification with Uncertainty Quantification
Università degli Studi di Milano, Department of Computer Science(米兰大学计算机科学系);Aix Marseille Univ, CNRS/IN2P3, CPPM(阿ix马赛大学,CNRS/IN2P3,CPPM);Université Côte d’Azur, INRIA, CNRS, Laboratoire J.A.Dieudonné, Maasai team(滨海大学,INRIA,CNRS,J.A.Dieudonné实验室,Maasai团队);Université Côte d’Azur, Observatoire de la Côte d’Azur, CNRS, Laboratoire Lagrange(滨海大学,滨海天文台,CNRS,Lagrange实验室);Université de Liège, STAR Institute(列日大学,STAR研究所)
Amazon Research Cambridge(亚马逊剑桥研究院);University of Cambridge(剑桥大学);Arrival(Arrival公司);University of Tübingen(图宾根大学);Facebook(Facebook公司);Microsoft Research Cambridge(微软剑桥研究院)
A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applications
College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机与软件工程学院);The Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen University(深圳大学广东省智能信息处理重点实验室);College of Mathematics and Statistics, Shenzhen University(深圳大学数学与统计学院);Department of Mathematics, The Chinese University of Hong Kong(香港中文大学数学系)
A Survey on Uncertainty Reasoning and Quantification for Decision Making: Belief Theory Meets Deep Learning
Virginia Tech(弗吉尼亚理工大学);University of Texas at Dallas(德克萨斯大学达拉斯分校);US Army Research Laboratory(美国陆军研究实验室);University of the District of Columbia(哥伦比亚特区大学);University of Oslo(奥斯陆大学)
A Comprehensive Survey on Evidential Deep Learning and Its Applications
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Peng Cheng Laboratory(鹏城实验室)
A Survey of Contextual Optimization Methods for Decision Making under Uncertainty
Department of Computer Science and Operations Research Université de Montréal(计算机科学与运筹学系,蒙特利尔大学);GERAD & Department of Decision Sciences HEC Montréal(GERAD与决策科学系,蒙特利尔HEC);CIRRELT & SCALE-AI Chair in Data-Driven Supply Chains, Department of Mathematical and Industrial Engineering, Polytechnique Montréal(CIRRELT与数据驱动供应链主席职位,数学与工业工程系,蒙特利尔Polytechnique)