arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1907.01995math.OCcs.LGstat.ML
收藏

随机多块ADMM在机器学习问题中的应用

On a Randomized Multi-Block ADMM for Solving Selected Machine Learning Problems

Stanford University(斯坦福大学);The University of Edinburgh(爱丁堡大学);Oracle Labs(甲骨文实验室)

Mingxi Zhu, Kresimir Mihic, Yinyu Ye

摘要:

乘法器的交替方向法(ADMM)由于其相对简单的特点,在解决大规模机器学习和信号处理问题中得到了广泛的关注。然而,经典ADMM的两块结构仍然限制了实际问题的规模。当通过变量分裂强制形成两块以上的结构时,收敛速度大大减慢,这与实际情况是一致的。最近,随机组装循环多块ADMM(RAC-MBADMM)的作者提出了解决一般的凸和非凸二次最优化问题,其中的块的数量可以去大于2,使每个子问题具有较小的规模,可以更有效地解决。在本文中,我们将这种方法应用于解决一些与凸二次优化相关的机器学习问题,如线性回归,LASSO,弹性网络和SVM。我们证明了该算法在标准的统计数据假设下线性收敛于期望。我们使用我们的通用求解器进行多个数值测试,解决合成和大规模的基准问题。我们的研究结果表明,RAC-MBADMM可以显着优于,在解决时间和质量,其他优化算法/代码来解决这些机器学习问题,并匹配的最佳定制的方法,如Glmnet或LIBSVM的性能。在某些问题区域,RAC-MBADMM甚至实现了优于定制方法的性能。

英文摘要:

The Alternating Direction Method of Multipliers (ADMM) has now days gained tremendous attentions for solving large-scale machine learning and signal processing problems due to the relative simplicity. However, the two-block structure of the classical ADMM still limits the size of the real problems being solved. When one forces a more-than-two-block structure by variable-splitting, the convergence speed slows down greatly as observed in practice. Recently, a randomly assembled cyclic multi-block ADMM (RAC-MBADMM) was developed by the authors for solving general convex and nonconvex quadratic optimization problems where the number of blocks can go greater than two so that each sub-problem has a smaller size and can be solved much more efficiently. In this paper, we apply this method to solving few selected machine learning problems related to convex quadratic optimization, such as Linear Regression, LASSO, Elastic-Net, and SVM. We prove that the algorithm would converge in expectation linearly under the standard statistical data assumptions. We use our general-purpose solver to conduct multiple numerical tests, solving both synthetic and large-scale bench-mark problems. Our results show that RAC-MBADMM could significantly outperform, in both solution time and quality, other optimization algorithms/codes for solving these machine learning problems, and match up the performance of the best tailored methods such as Glmnet or LIBSVM. In certain problem regions RAC-MBADMM even achieves a superior performance than that of the tailored methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑