基于加权最小二乘的降秩矩阵逼近问题的变量投影框架
Variable projection framework for the reduced-rank matrix approximation problem by weighted least-squares
AI总结:
本文系统回顾并发展了用于加权低秩逼近问题的变量投影二阶优化算法,推导了雅可比矩阵和海森矩阵的新公式并揭示其秩亏性质,同时建立了变量投影框架与Grassmann流形上黎曼优化的联系,为设计更鲁棒高效的WLRA求解算法提供了理论基础。
摘要:
在本专著中,我们回顾并发展了用于加权低秩逼近(WLRA)问题的变量投影高斯-牛顿法、Levenberg-Marquardt法和牛顿法,该问题如今在众多科学领域中的应用日益增多。我们特别关注这些变量投影二阶算法在鲁棒性、效率和可扩展性方面的表现,使其能够用于更大规模的数据集。目前许多实际问题中常见的大数据集,以往只能采用基于局部优化顺序重复的一阶算法(例如majorization、Expectation-Maximization或交替最小二乘法),或梯度下降的变体(例如共轭梯度法、近端梯度法或随机梯度下降法),或由这两类方法组合而成的混合算法,因为这些方法每次迭代的成本和内存需求较低。在回顾变量投影算法的同时,我们为这些变量投影方法中涉及的雅可比矩阵和海森矩阵推导了新的公式,并展示了它们非常特殊的性质,例如雅可比矩阵的一致秩亏,以及在与WLRA问题相关的代价函数的(局部)极小值处海森矩阵的秩亏。这些系统性的秩亏必须在算法的任何实际实现中加以考虑。这些不同的性质以及WLRA问题非常特殊的几何结构在过去未被充分认识,一直是开发用于求解WLRA问题的鲁棒变量投影二阶算法的主要障碍。此外,我们证明了变量投影框架为WLRA问题的可解性、景观结构(landscape)和非光滑性提供了独到的见解。它还有助于描述此前互不相关的、为求解该问题而提出的方法之间的紧密联系。具体而言,我们阐明了变量投影框架与用于WLRA问题的Grassmann流形上的黎曼优化之间的密切联系。我们期望计算机视觉、信号处理、推荐系统、机器学习、多元统计学和地球物理科学等不同领域的软件开发者和实践者能够从本专著的结果中受益,从而设计出更鲁棒、更精确的算法来求解WLRA问题。
英文摘要:
In this monograph, we review and develop variable projection Gauss-Newton, Levenberg-Marquardt and Newton methods for the Weighted Low-Rank Approximation (WLRA) problem, which has now an increasing number of applications in many scientific fields. Particular attention is drawn at the robustness, efficiency and scalability of these variable projection second-order algorithms such that they can be used also on larger datasets now commonly found in many practical problems for which only first-order algorithms based on sequential repetitions of local optimization (e.g., majorization, Expectation-Maximization or alternating least-squares methods) or variations of gradient descent (e.g., conjugate, proximal or stochastic gradient descent methods), or hybrid algorithms from these two classes of methods, were only feasible due to their lower cost and memory requirement per iteration. In parallel with this review of variable projection algorithms, we develop new formulae for the Jacobian and Hessian matrices involved in these variable projection methods and demonstrate their very specific properties such as the uniform rank deficiency of the Jacobian matrix or the rank deficiency of the Hessian matrix at the (local) minimizers of the cost function associated with the WLRA problem. These systematic deficiencies must be taken into account in any practical implementations of the algorithms. These different properties and the very particular geometry of the WLRA problem have not been well appreciated in the past and have been the main obstacles in the development of robust variable projection second-order algorithms for solving the WLRA problem. In addition, we demonstrate that the variable projection framework gives original insights on the solvability, the landscape and the non-smoothness of the WLRA problem. It also helps to describe the tight links between previously unrelated methods, which have been proposed to solve it. Specifically, we illustrate the closed links between the variable projection framework and Riemannian optimization on the Grassmann manifold for the WLRA problem. We expect that software's developers and practitioners in different fields such as computer vision, signal processing, recommender systems, machine learning, multivariate statistics and geophysical sciences will benefit from the results in this monograph in order to devise more robust and accurate algorithms to solve the WLRA problem.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。