采用自适应采样的 L-SVRG 与 L-Katyusha
L-SVRG and L-Katyusha with Adaptive Sampling
AI总结：
本文针对非均匀采样依赖光滑性常数先验的问题，为 L-SVRG 与 L-Katyusha 提出低开销、随迭代更新的自适应采样策略，并证明凸情形下的收敛性与性能优势。
摘要：
基于随机梯度的优化方法，例如 L-SVRG 及其加速变体 L-Katyusha（Kovalev 等人，2020），被广泛用于训练机器学习模型。通过从非均匀分布中采样观测值，可以提升 L-SVRG 和 L-Katyusha 在理论与实证方面的性能（Qian 等人，2021）。然而，设计理想的采样分布需要预先知道光滑性常数；当模型参数维度较高时，这在实践中可能因计算上不可行而难以获得。为解决该问题，我们为 L-SVRG 和 L-Katyusha 提出了一种自适应采样策略，能够以很小的计算开销学习采样分布，并允许该分布随迭代过程变化，同时不需要任何关于问题参数的先验知识。我们证明了当采样分布随迭代变化时，L-SVRG 和 L-Katyusha 对凸目标函数的收敛保证。结果表明，即使没有先验信息，所提出的自适应采样策略也能达到、在某些情况下甚至超过 Qian 等人（2021）采样方案的性能。大量仿真支持了我们的理论以及所提采样方案在真实数据上的实用价值。
英文摘要：
Stochastic gradient-based optimization methods, such as L-SVRG and its accelerated variant L-Katyusha (Kovalev et al., 2020), are widely used to train machine learning models.The theoretical and empirical performance of L-SVRG and L-Katyusha can be improved by sampling observations from a non-uniform distribution (Qian et al., 2021). However,designing a desired sampling distribution requires prior knowledge of smoothness constants, which can be computationally intractable to obtain in practice when the dimension of the model parameter is high. To address this issue, we propose an adaptive sampling strategy for L-SVRG and L-Katyusha that can learn the sampling distribution with little computational overhead, while allowing it to change with iterates, and at the same time does not require any prior knowledge of the problem parameters. We prove convergence guarantees for L-SVRG and L-Katyusha for convex objectives when the sampling distribution changes with iterates. Our results show that even without prior information, the proposed adaptive sampling strategy matches, and in some cases even surpasses, the performance of the sampling scheme in Qian et al. (2021). Extensive simulations support our theory and the practical utility of the proposed sampling scheme on real data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
