具有次线性计算复杂度的在线核回归近优算法
Nearly Optimal Algorithms with Sublinear Computational Complexity for Online Kernel Regression
- Tianjin University（天津大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
针对在线核回归中遗憾界与计算成本的权衡难题，提出AOGD-ALD和NONS-ALD两种算法，通过动态维护近正交基近似核映射，在次线性复杂度下实现两类近优遗憾界。
AI中文摘要：
遗憾界与计算成本之间的权衡是在线核回归的一个基本问题，此前针对该权衡开展研究的算法无法在次线性计算复杂度下保持最优遗憾界。本文提出了两种新算法AOGD-ALD和NONS-ALD，它们能在次线性计算复杂度下保持近优遗憾界，并给出了算法适用的充分条件。两种算法都动态维护一组用于近似核映射的近正交基，并通过控制近似误差来保持近优遗憾界，基的数量取决于近似误差和核矩阵特征值的衰减速率。若特征值呈指数衰减，则AOGD-ALD和NONS-ALD分别达到$O(\sqrt{L(f)})$和$O(\mathrm{d}_{\mathrm{eff}}(μ)\ln{T})$的遗憾界，计算复杂度为$O(\ln^2{T})$。若特征值以次数$p\geq 1$的多项式速率衰减，则两种算法分别在$p>4$和$p\geq 10$的情况下，以$o(T)$的计算复杂度保持相同的遗憾界。其中$L(f)$是$f$的累积损失，$\mathrm{d}_{\mathrm{eff}}(μ)$是问题的有效维度。这两种遗憾界均为近优，且二者不具有可比性。
英文摘要：
The trade-off between regret and computational cost is a fundamental problem for online kernel regression, and previous algorithms worked on the trade-off can not keep optimal regret bounds at a sublinear computational complexity. In this paper, we propose two new algorithms, AOGD-ALD and NONS-ALD, which can keep nearly optimal regret bounds at a sublinear computational complexity, and give sufficient conditions under which our algorithms work. Both algorithms dynamically maintain a group of nearly orthogonal basis used to approximate the kernel mapping, and keep nearly optimal regret bounds by controlling the approximate error. The number of basis depends on the approximate error and the decay rate of eigenvalues of the kernel matrix. If the eigenvalues decay exponentially, then AOGD-ALD and NONS-ALD separately achieves a regret of $O(\sqrt{L(f)})$ and $O(\mathrm{d}_{\mathrm{eff}}(μ)\ln{T})$ at a computational complexity in $O(\ln^2{T})$. If the eigenvalues decay polynomially with degree $p\geq 1$, then our algorithms keep the same regret bounds at a computational complexity in $o(T)$ in the case of $p>4$ and $p\geq 10$, respectively. $L(f)$ is the cumulative losses of $f$ and $\mathrm{d}_{\mathrm{eff}}(μ)$ is the effective dimension of the problem. The two regret bounds are nearly optimal and are not comparable.
