Localized sketching for matrix multiplication and ridge regression

Newton-LESS: Sparsification without Trade-offs for the Sketched Newton Update

In second-order optimization, a potential bottleneck can be computing the Hessian matrix of the optimized function at every iteration. Randomized sketching has emerged as a powerful technique for constructing estimates of the Hessian which can be used to perform approximate Newton steps. This involves multiplication by a random sketching matrix, which introduces a trade-off between the computational cost of sketching and the convergence rate of the optimization algorithm. A theoretically desirable but practically much too expensive choice is to use a dense Gaussian sketching matrix, which prod…

Ridge Regression: Structure, Cross-Validation, and Sketching

We study the following three fundamental problems about ridge regression: (1) what is the structure of the estimator? (2) how to correctly use cross-validation to choose the regularization parameter? and (3) how to accelerate computation without losing too much accuracy? We consider the three problems in a unified large-data linear model. We give a precise representation of ridge regression as a covariance matrix-dependent linear combination of the true parameter and the noise. We study the bias of $K$-fold cross-validation for choosing the regularization parameter, and propose a simple bias-c…

Learning the Positions in CountSketch

We consider sketching algorithms which first quickly compress data by multiplication with a random sketch matrix, and then apply the sketch to quickly solve an optimization problem, e.g., low rank approximation. In the learning-based sketching paradigm proposed by Indyk et al. [2019], the sketch matrix is found by choosing a random sparse matrix, e.g., the CountSketch, and then updating the values of the non-zero entries by running gradient descent on a training data set. Despite the growing body of work on this paradigm, a noticeable omission is that the locations of the non-zero entries of p…

学习CountSketch中的位置

Learning the Positions in CountSketch

针对现有基于学习的草图算法仅优化非零元素取值、位置固定的不足，提出同时优化非零元素位置的算法，解决贪心算法训练慢的问题，实现低秩逼近和海森矩阵逼近的高精度快速求解，少量训练数据下仍能显著降误差。

草图化无岭线性回归：下采样的作用

Sketched Ridgeless Linear Regression: The Role of Downsampling

本文提出过参数化的对偶视角，证明在比例区间下采样可改善泛化，确定最优草图大小并给出经验确定方法，理论经数值验证。

渐近自由的素描岭集成：风险、交叉验证与调参

Asymptotically free sketched ridge ensembles: Risks, cross-validation, and tuning

本文利用随机矩阵理论证明了广义交叉验证在估计素描岭回归集成预测风险时的一致性，提出风险分解与扩展估计方法，并证明仅调素描大小即可全局优化风险，同时提出集成技巧以高效估计未素描岭回归风险。