This work considers methods for imposing sparsity in Bayesian regression with applications in nonlinear system identification. We first review automatic relevance determination (ARD) and analytically demonstrate the need to additional regularization or thresholding to achieve sparse models. We then discuss two classes of methods, regularization based and thresholding based, which build on ARD to learn parsimonious solutions to linear problems. In the case of orthogonal covariates, we analytically demonstrate favorable performance with regards to learning a small set of active terms in a linear system with a sparse solution. Several example problems are presented to compare the set of proposed methods in terms of advantages and limitations to ARD in bases with hundreds of elements. The aim of this paper is to analyze and understand the assumptions that lead to several algorithms and to provide theoretical and empirical results so that the reader may gain insight and make more informed choices regarding sparse Bayesian regression.
Towards Compact Neural Networks via End-to-End Training: A Bayesian Tensor Approach with Automatic Rank Determination
University of California Santa Barbara（加州大学圣巴巴拉分校）；Department of Mathematics, University of California Santa Barbara（加州大学圣巴巴拉分校数学系）；Department of Electrical and Computer Engineering, University of California Santa Barbara（加州大学圣巴巴拉分校电气与计算机工程系）；Facebook AI Systems Hardware/Software Co-design（Facebook人工智能系统硬件/软件协同设计）
虽然后训练模型压缩可以大大降低深度神经网络的推理成本，但未压缩的训练仍然会消耗大量的硬件资源、运行时间和能量。非常希望以低存储器和低计算成本从头开始直接训练紧凑的神经网络。低秩张量分解是降低大规模神经网络存储和计算需求的有效方法之一。然而，直接训练一个低秩张量化神经网络是一个非常具有挑战性的任务，因为很难确定一个合适的张量秩{ \it a priori}，它控制着训练过程中模型的复杂度和压缩比。本文提出了一种新的端到端的框架低秩张量训练的神经网络。我们首先开发了一个灵活的贝叶斯模型，可以处理各种低秩张量格式（例如，CP、Tucker、张量训练和张量-训练矩阵），其在训练中压缩神经网络参数。该模型可以自动确定非线性正演模型中张量的秩，这是现有贝叶斯张量方法无法实现的。我们进一步开发了一个可扩展的随机变分推理求解器来估计训练中大规模问题的后验密度。我们的工作为端到端张量化训练提供了第一个通用的秩自适应框架。我们对各种神经网络架构的数值结果表明，在训练过程中，参数的数量级减少和精度损失很小（甚至更好的精度）。具体来说，在具有超过$4.2\times 10^9$模型参数的非常大的深度学习推荐系统上，我们的方法可以在训练过程中自动将变量减少到只有$1.6\times 10^5$（即，$2.6\times 10^4$倍），同时达到几乎相同的精度。
School of Data Science, Fudan University, Shanghai, China（复旦大学数据科学学院）；Incubation Institute, Fudan University, Shanghai, China（复旦大学孵化院）；Shanghai Academy of Artificial Intelligence for Science, Shanghai, China（上海人工智能科学研究院）；Eberly College of Science, Pennsylvania State University, PA, United State（宾夕法尼亚州立大学科学学院）；Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, United Kingdom（剑桥大学应用数学与理论物理系）；Zhongshan Hospital, Fudan University, Shanghai, China（复旦大学中山医院）
