arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2108.08605cs.LG
收藏

基于对数线性复杂度多层循环矩阵求解KLR的快速牛顿法

Fast Newton method solving KLR based on Multilevel Circulant Matrix with log-linear complexity

Xidian University(西安电子科技大学)

Junna Zhang, Shuisheng Zhou, Cui Fu, Feng Ye

摘要:

核逻辑回归(KLR)是机器学习中常用的非线性分类器。随着数据量的爆炸式增长,大型稠密核矩阵的存储和计算是扩展KLR的主要挑战。即使采用Nyström近似求解KLR,也面临着$O(nc^2)$的时间复杂度和$O(nc)$的空间复杂度,其中$n$是训练实例的数量,$c$是采样大小。本文利用多级循环矩阵的存储和计算优势,提出了一种快速牛顿法,有效地求解大规模KLR问题。具体地,通过用MCM近似核矩阵,存储空间减少到$O(n)$,并且进一步将牛顿方程的系数矩阵近似为MCM,牛顿迭代的计算复杂性减少到$O(n \log n)$。由于MCM(或其逆)与向量的乘法可以用多维快速傅立叶变换(mFFT)实现,因此该方法的每次迭代时间复杂度为对数线性。对一些大规模二分类和多分类问题的实验结果表明,该方法使KLR能够在不牺牲测试准确性的情况下以更少的内存消耗和更少的训练时间扩展到大规模问题。

英文摘要:

Kernel logistic regression (KLR) is a conventional nonlinear classifier in machine learning. With the explosive growth of data size, the storage and computation of large dense kernel matrices is a major challenge in scaling KLR. Even the nyström approximation is applied to solve KLR, it also faces the time complexity of $O(nc^2)$ and the space complexity of $O(nc)$, where $n$ is the number of training instances and $c$ is the sampling size. In this paper, we propose a fast Newton method efficiently solving large-scale KLR problems by exploiting the storage and computing advantages of multilevel circulant matrix (MCM). Specifically, by approximating the kernel matrix with an MCM, the storage space is reduced to $O(n)$, and further approximating the coefficient matrix of the Newton equation as MCM, the computational complexity of Newton iteration is reduced to $O(n \log n)$. The proposed method can run in log-linear time complexity per iteration, because the multiplication of MCM (or its inverse) and vector can be implemented the multidimensional fast Fourier transform (mFFT). Experimental results on some large-scale binary-classification and multi-classification problems show that the proposed method enables KLR to scale to large scale problems with less memory consumption and less training time without sacrificing test accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑