Training neural networks on randomly generated artificial datasets yields Bayesian models that capture the prior defined by the dataset-generating distribution. Prior-data Fitted Networks (PFNs) are a class of methods designed to leverage this insight. In an era of rapidly increasing computational resources for pre-training and a near stagnation in the generation of new real-world data in many applications, PFNs are poised to play a more important role across a wide range of applications. They enable the efficient allocation of pre-training compute to low-data scenarios. Originally applied to small Bayesian modeling tasks, the field of PFNs has significantly expanded to address more complex domains and larger datasets. This position paper argues that PFNs and other amortized inference approaches represent the future of Bayesian inference, leveraging amortized learning to tackle data-scarce problems. We thus believe they are a fruitful area of research. In this position paper, we explore their potential and directions to address their current limitations.
Survival In-Context: Amortized Bayesian Survival Analysis via Prior-Fitted Networks
Institute of Diagnostic and Interventional Radiology, Technical University of Munich, Germany(慕尼黑技术大学诊断与介入放射学研究所,德国);Chair for AI in Healthcare and Medicine, Technical University of Munich, Germany(慕尼黑技术大学医疗人工智能与医学研究所,德国);Hasso Plattner Institute for Digital Engineering, University of Potsdam, Germany(波茨坦大学数字工程哈索普兰特纳研究所,德国);University Hospital Hamburg-Eppendorf, Germany(汉堡-埃彭多夫大学医院,德国);Department of Computing, Imperial College London, UK(伦敦帝国理工学院计算系,英国);Munich Center for Machine Learning (MCML), Germany(慕尼黑机器学习中心(MCML),德国)
University of Dublin, Trinity College(都柏林大学,三一学院);IBM;Gatsby Computational Neuroscience Unit, University College London(大学学院伦敦的加布里埃尔计算神经科学单位);Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI), Saarbrücken, Germany(德国萨尔布吕肯德意志人工智能研究中心(DFKI));Department of Philosophy, University of Bergen(卑尔根大学哲学系)
贝叶斯神经网络利用概率层来捕获权重和激活的不确定性,并使用贝叶斯推理进行训练。由于这些概率层被设计成其确定性对应部分的直接替代,贝叶斯神经网络提供了一种直接和自然的方式来扩展传统的深度神经网络以支持概率深度学习。然而,由于贝叶斯神经网络的复杂性,理解、设计和训练贝叶斯神经网络是非常重要的。我们讨论了贝叶斯神经网络的本质,包括对偶(深度神经网络,概率模型),近似贝叶斯推理,贝叶斯先验,贝叶斯后验和深度变分学习。我们使用TensorFlow Probability API和代码示例进行说明。贝叶斯神经网络的主要问题是,深度神经网络的架构使得考虑大量连续层的不确定性变得非常冗余和昂贵。混合贝叶斯神经网络,它使用很少的概率层,公正地定位在网络中,提供了一个实用的解决方案。
Longitudinal Bayesian Learning of Continuous Disease Position across the Alzheimer's Disease Continuum
University of Texas Rio Grande Valley(德克萨斯大学里奥格兰德河谷分校);University of Pittsburgh(匹兹堡大学);Eli and Lilly Company(礼来公司);Washington University in St. Louis(圣路易斯华盛顿大学)
State-Space Models for Tabular Prior-Data Fitted Networks
L3S Research Center, Leibniz University Hannover, Hannover, Germany(L3S研究所以及汉诺威莱布尼茨大学);University of Applied Sciences Rosenheim, Rosenheim, Germany(罗斯海姆应用科学大学);Technical University of Munich, Munich, Germany(慕尼黑技术大学)
Position: agentic AI orchestration should be Bayes-consistent
ESSEC Business School(ESSEC商学院);VinUniversity(文大学);Imperial College London(伦敦帝国理工学院);Mila - Quebec AI Institute(魁北克人工智能研究所);Technical University of Denmark(丹麦技术大学);Pyramidal Inc.(Pyramidal公司);University of Notre Dame(诺特大学);Cognizant AI Lab(Cognizant人工智能实验室);University College London(伦敦大学学院);KTH Royal Institute of Technology(瑞典皇家理工学院);Ghent University(根特大学);New York University(纽约大学)
Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI
The University of Manchester(曼彻斯特大学);Broad Institute of MIT and Harvard(麻省理工学院与哈佛大学博德研究所);Spotify(Spotify公司);University of Bristol(布里斯托尔大学);Centre Inria de l’Université Grenoble Alpes(格勒诺布尔阿尔卑斯大学法国国家信息与自动化研究所中心);Duke University(杜克大学);KAUST(阿卜杜拉国王科技大学);Helmholtz AI(亥姆霍兹人工智能研究所);Technical University of Munich(慕尼黑工业大学);Munich Center for Machine Learning(慕尼黑机器学习中心);University of Tübingen(蒂宾根大学);University of Cambridge(剑桥大学);University of Oslo(奥斯陆大学);Norwegian University of Life Sciences(挪威生命科学大学);ETH Zurich(苏黎世联邦理工学院);Chan Zuckerberg Initiative(陈·扎克伯格倡议);RIKEN(理化学研究所)
本文论证贝叶斯深度学习能弥补大规模 AI 对不确定性、主动与持续学习及科学数据等问题的忽视,并展望其与基础模型结合的研究方向。