在较弱条件下具有改进遗憾界的在线稀疏线性回归多项式时间算法
A Polynomial-time Algorithm for Online Sparse Linear Regression with Improved Regret Bound under Weaker Conditions
- Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
- Tianjin University（天津大学）
- Fudan University（复旦大学）
- Shanghai Academy of AI for Science（上海科学智能研究院）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本文研究了在线稀疏线性回归问题，提出了一种新的多项式时间算法，在较弱的兼容性条件下利用Dantzig Selector及多项新技术，显著改进了先前的遗憾界，并扩展至额外观测场景。
AI中文摘要：
本文研究了在线稀疏线性回归（OSLR）问题，其中算法被限制为每个实例仅访问 $d$ 个属性中的 $k$ 个用于预测，该问题已被证明是 NP-hard 的。先前的工作给出了多项式时间算法，假设数据矩阵满足特征线性无关性、兼容性条件或受限等距性。我们引入了一种新的多项式时间算法，在比其他两个假设更弱的兼容性条件下，显著改进了先前的遗憾界（Ito et al., 2017）。这些改进得益于我们的估计器具有更紧的 $\ell_1$-范数误差收敛率。我们的算法利用了被广泛研究的 Dantzig Selector，但重要的是引入了几种新技术，包括用于估计协方差矩阵的依赖算法的采样方案、自适应参数调优方案，以及带有精心初始化的批量在线牛顿步。我们还给出了新颖且非平凡的分析，包括用于分析 $\ell_1$-范数误差的归纳法、对非独立随机变量协方差的仔细分析，以及对遗憾的分解。我们进一步将算法扩展到具有额外观测的 OSLR，其中算法可以在每次预测后观察额外的 $k_0$ 个属性，并改进了先前的遗憾界（Kale et al., 2017; Ito et al., 2017）。
英文摘要：
In this paper, we study the problem of online sparse linear regression (OSLR) where the algorithms are restricted to accessing only $k$ out of $d$ attributes per instance for prediction, which was proved to be NP-hard. Previous work gave polynomial-time algorithms assuming the data matrix satisfies the linear independence of features, the compatibility condition, or the restricted isometry property. We introduce a new polynomial-time algorithm, which significantly improves previous regret bounds (Ito et al., 2017) under the compatibility condition that is weaker than the other two assumptions. The improvements benefit from a tighter convergence rate of the $\ell_1$-norm error of our estimators. Our algorithm leverages the well-studied Dantzig Selector, but importantly with several novel techniques, including an algorithm-dependent sampling scheme for estimating the covariance matrix, an adaptive parameter tuning scheme, and a batching online Newton step with careful initializations. We also give novel and non-trivial analyses, including an induction method for analyzing the $\ell_1$-norm error, careful analyses on the covariance of non-independent random variables, and a decomposition on the regret. We further extend our algorithm to OSLR with additional observations where the algorithms can observe additional $k_0$ attributes after each prediction, and improve previous regret bounds (Kale et al., 2017; Ito et al., 2017).
