用于监督机器学习的Optimization:数据和参数的随机化算法
Optimization for Supervised Machine Learning: Randomized Algorithms for Data and Parameters
摘要:
机器学习和数据科学中的许多关键问题通常被建模为最优化问题,并通过优化算法来解决。随着数据量的增加以及用于制定这些经常病态的优化任务的统计模型的大小和复杂性,需要能够应对这些挑战的新的高效算法。在这篇论文中,我们以不同的方式处理每一个困难的来源。为了有效地解决大数据问题,我们开发了新的方法,在每次迭代中只检查训练数据的一个小的随机子集。为了处理大模型问题,我们开发了在每次迭代中仅更新模型参数的随机子集的方法。最后,为了处理病态问题,我们设计的方法,将高阶信息或Nesterov的加速度/动量。在所有情况下,随机性都被视为一种强大的算法工具,我们可以在理论和实验中进行调整,以获得最佳结果。我们的算法主要应用于通过正则化经验风险最小化来训练监督机器学习模型,这是训练此类模型的主要范例。然而,由于其通用性,我们的方法可以应用于许多其他领域,包括但不限于数据科学,工程,科学计算和统计。
英文摘要:
Many key problems in machine learning and data science are routinely modeled as optimization problems and solved via optimization algorithms. With the increase of the volume of data and the size and complexity of the statistical models used to formulate these often ill-conditioned optimization tasks, there is a need for new efficient algorithms able to cope with these challenges. In this thesis, we deal with each of these sources of difficulty in a different way. To efficiently address the big data issue, we develop new methods which in each iteration examine a small random subset of the training data only. To handle the big model issue, we develop methods which in each iteration update a random subset of the model parameters only. Finally, to deal with ill-conditioned problems, we devise methods that incorporate either higher-order information or Nesterov's acceleration/momentum. In all cases, randomness is viewed as a powerful algorithmic tool that we tune, both in theory and in experiments, to achieve the best results. Our algorithms have their primary application in training supervised machine learning models via regularized empirical risk minimization, which is the dominant paradigm for training such models. However, due to their generality, our methods can be applied in many other fields, including but not limited to data science, engineering, scientific computing, and statistics.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。