Revisiting the Nystrom Method for Improved Large-Scale Machine Learning
California Institute of Technology(加州理工学院);Stanford University(斯坦福大学)
Alex Gittens, Michael W. Mahoney
备注:
60 pages, 15 color figures; updated proof of Frobenius norm bounds, added comparison to projection-based low-rank approximations, and an analysis of the power method applied to SPSD sketches
We reconsider randomized algorithms for the low-rank approximation of symmetric positive semi-definite (SPSD) matrices such as Laplacian and kernel matrices that arise in data analysis and machine learning applications. Our main results consist of an empirical evaluation of the performance quality and running time of sampling and projection methods on a diverse suite of SPSD matrices. Our results highlight complementary aspects of sampling versus projection methods; they characterize the effects of common data preprocessing steps on the performance of these algorithms; and they point to important differences between uniform sampling and nonuniform sampling methods based on leverage scores. In addition, our empirical results illustrate that existing theory is so weak that it does not provide even a qualitative guide to practice. Thus, we complement our empirical results with a suite of worst-case theoretical bounds for both random sampling and random projection methods. These bounds are qualitatively superior to existing bounds---e.g. improved additive-error bounds for spectral and Frobenius norm error and relative-error bounds for trace norm error---and they point to future directions to make these algorithms useful in even larger-scale machine learning applications.
Revisiting the Bethe-Hessian: Improved Community Detection in Sparse Heterogeneous Graphs
GIPSA-lab(GIPSA实验室);UGA(格勒诺布尔阿尔卑斯大学);CNRS(法国国家科学研究中心);Grenoble INP(格勒诺布尔国立高等物理、数学与信息学院);L2S(信号与系统实验室);CentraleSupélec(中央理工-高等电力学院);University of Paris Saclay(巴黎萨克雷大学)
Determinantal Point Processes in Randomized Numerical Linear Algebra
University of California, Berkeley(加州大学伯克利分校);ICSI(国际计算机科学研究所)
Randomized Numerical Linear Algebra(RandNLA)使用随机性来开发用于科学计算、数据科学、机器学习等中出现的矩阵问题的改进算法。行列式点过程(DPPs)是一类随机点过程,其概率分布的特征在于核矩阵的子行列式。最近的工作揭示了DPP和RandNLA之间的深刻而富有成效的联系,这导致了新的保证和改进的算法,这两个领域都感兴趣。我们提供了一个概述这一令人兴奋的新的研究路线,包括简要介绍RandNLA和DPP,以及DPP的经典线性代数任务,如最小二乘回归,低秩近似和Nyström方法的应用。例如,使用DPP的随机抽样导致了最小二乘的新的无偏估计量,使这些算法的统计和推理理解更加精确; DPP在某种意义上是Nyström方法的最佳随机算法;而称为杠杆分数抽样的RandNLA技术可以作为DPP的边际分布而导出。我们还讨论了最近的算法的发展,说明,虽然不是很有效的标准RandNLA技术,基于DPP的算法只是适度更昂贵。
Revisiting Co-Occurring Directions: Sharper Analysis and Efficient Algorithm for Sparse Matrices
The Hong Kong University of Science and Technology(香港科技大学);Shanghai Jiao Tong University(上海交通大学);Peking University(北京大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
Revisiting Recursive Least Squares for Training Deep Neural Networks
Hainan University(海南大学);Yangtze Normal University(长江师范学院)
递归最小二乘(Recursive Least Squares,RLS)算法由于其快速收敛性曾被广泛用于训练小规模神经网络。然而,以前的RLS算法不适合训练深度神经网络(DNN),因为它们具有高计算复杂性和太多的先决条件。为了克服这些缺点,本文提出了三种新的RLS优化算法,分别用于训练前馈神经网络、卷积神经网络和递归神经网络(包括长短期记忆网络),算法采用误差反向传播和我们的平均逼近RLS方法,并结合线性最小二乘损失函数关于隐层线性输出的等效梯度。与以往的RLS优化算法相比,我们的算法是简单和优雅的。它们可以被看作是一种改进的随机梯度下降(SGD)算法,它使用每层的逆自相关矩阵作为自适应学习率。它们的时间和空间复杂度仅是SGD的几倍。它们只要求损失函数是均方误差,输出层的激活函数是可逆的。事实上,我们的算法也可以与其他一阶优化算法结合使用,而不需要这两个先决条件。此外,我们提出了两个改进的方法,我们的算法。最后,我们在MNIST,CIFAR-10和IMDB数据集上证明了它们与Adam算法的有效性,并通过实验研究了它们的超参数的影响。