基于非利普希茨符号函数的分布式优化与机器学习：收敛速率与最优性间隙的权衡

中文摘要 AI 辅助

近年来，大规模数据集的普及以及对复杂学习模型的需求，促使高效分布式机器学习（ML）解决方案的开发。收敛速度是影响这些分布式框架实用性和有效性的关键因素。近期，非利普希茨连续优化算法被提出，用于改善现有线性解决方案收敛速率缓慢的问题。符号函数此前已在共识与控制文献中被研究，以在规定时间内实现快速收敛，并为含噪声/离群值的数据提供鲁棒算法。然而，本文研究表明，在离散时间设定下，这些算法会导致目标函数出现最优性间隙和稳态残差。这促使我们从收敛速率与最优性间隙的权衡角度研究分布式优化与ML算法。在此方向上，我们专门考虑分布式回归问题，并通过应用线性和非利普希茨符号函数两种方式检验其收敛速率。我们通过大量仿真验证了所提分布式回归方法，结果显示，尽管采用符号函数可实现更快收敛，但会产生较大的最优性间隙。本文的研究成果可能有助于推进类似分布式算法的相关讨论，例如分布式约束优化和分布式估计等领域。

英文摘要

In recent years, the prevalence of large-scale data-sets and the demand for sophisti-cated learning models have necessitated the development of efficient distributed ma-chine learning (ML) solutions. Convergence speed is a critical factor influencing the practicality and effectiveness of these distributed frameworks. Recently, non-Lipschitz continuous optimization algorithms have been proposed to improve the slow conver-gence rate of the existing linear solutions. The use of signum-based functions is previ-ously considered in consensus and control literature to reach fast convergence in the prescribed time and also to provide robust algorithms to noisy/outlier data. However, as shown in this work, these algorithms lead to an optimality gap and steady-state re-sidual of the objective function in discrete-time setup. This motivates us to investigate the distributed optimization and ML algorithms in terms of trade-off between conver-gence rate and optimality gap. In this direction, we specifically consider the distributed regression problem and check its convergence rate by applying both linear and non-Lipschitz signum-based functions. We check our distributed regression approach by extensive simulations. Our results show that although adopting signum-based func-tions may give faster convergence, it results in large optimality gaps. The findings pre-sented in this paper may contribute to and advance the ongoing discourse of similar distributed algorithms, e.g., for distributed constrained optimization and distributed estimation.

发表机构

• Semnan University（塞姆南大学）
• Oregon State University（俄勒冈州立大学）
• Kazan Federal University（喀山联邦大学）
• Sharif University of Technology（谢里夫理工大学）

机构由 AI 辅助整理，请以论文原文为准。