Normalising flows are tractable probabilistic models that leverage the power of deep learning to describe a wide parametric family of distributions, all while remaining trainable using maximum likelihood. We discuss how these methods can be further improved based on insights from robust (in particular, resistant) statistics. Specifically, we propose to endow flow-based models with fat-tailed latent distributions such as multivariate Student's $t$, as a simple drop-in replacement for the Gaussian distribution used by conventional normalising flows. While robustness brings many advantages, this paper explores two of them: 1) We describe how using fatter-tailed base distributions can give benefits similar to gradient clipping, but without compromising the asymptotic consistency of the method. 2) We also discuss how robust ideas lead to models with reduced generalisation gap and improved held-out data likelihood. Experiments on several different datasets confirm the efficacy of the proposed approach in both regards.
You Are What You Eat -- AI Alignment Requires Understanding How Data Shapes Structure and Generalisation
University College London（伦敦大学学院）；University of Amsterdam（阿姆斯特丹大学）；University of Oxford（牛津大学）；Monash University（莫纳什大学）；the University of Melbourne（墨尔本大学）；Gradient Institute（格雷迪恩研究所）
Are nuclear masks all you need for improved out-of-domain generalisation? A closer look at cancer classification in histopathology
University of Oslo（奥斯陆大学）；Otto-von-Guericke University Magdeburg（马格德堡奥托·冯·格里克大学）；Singapore Institute of Technology（新加坡科技学院）；Oslo University Hospital（奥斯陆大学医院）；UiT The Arctic University of Norway（挪威北极大学）
Accuracy on the wrong line: On the pitfalls of noisy data for out-of-distribution generalisation
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of California, Berkeley（加州大学伯克利分校）；Halıcıoğlu Data Science Institute, University of California San Diego（加州大学圣迭戈分校哈利西奥格鲁数据科学学院）；University of Michigan, Ann Arbor（密歇根大学安娜堡分校）
The Why, What and How of Artificial General Intelligence Chip Development
Kerala University of Digital Sciences, Innovation and Technology (KUDSIT)（喀拉拉邦数字科学、创新与技术大学）；Indian Institute of Information Technology and Management, Kerala（喀拉拉邦印度信息技术与管理学院）
A Stochastic Bundle Method for Interpolating Networks
University of Oxford（牛津大学）；Toshiba Europe Ltd（东芝欧洲有限公司）
我们提出了一种新的方法来训练能够插值的深度神经网络，即将经验损失驱动到零。在每一次迭代中，我们的方法构造了学习目标的随机近似。近似，称为丛，是线性函数的逐点最大值。我们的捆绑包包含一个常数函数，它降低了经验损失。这使我们能够计算自动自适应学习率，从而提供准确的解决方案。此外，我们的捆绑包还包括在当前时间点计算的线性近似和DNN参数的其他线性估计。这些额外近似的使用使我们的方法对其超参数更加鲁棒。基于其理想的经验性质，我们将我们的方法称为Bundle Optimisation for Robust and Accurate Training（BORAT）。为了操作BORAT，我们设计了一种新的算法，有效地优化束近似在每次迭代。我们建立了理论上的收敛BORAT在凸和非凸设置。使用标准的公开数据集，我们提供了一个彻底的比较BORAT其他单超参数优化算法。我们的实验表明，BORAT匹配这些方法的最先进的泛化性能，是最强大的。
