利用目标函数阶数$p>2$的导数的优化方法被称为张量方法。其中,每一步最小化正则化$p$阶泰勒展开的方法已被证明具有最优全局复杂度,且复杂度随$p$增大而提升。Doikov和Nesterov[Math. Program., 193 (2022), pp. 315--336]研究了这类优化算法在具有Lipschitz连续$p$阶导数、且为$q$阶一致凸的函数上的局部收敛性。我们将这些局部收敛结果扩展到局部一致凸函数与无需已知Lipschitz常数的完全自适应方法,从而首次给出了AR$p$的精确局部收敛速率。我们讨论了非凸局部模型和模型极小值不唯一所带来的出人意料的新挑战。对于$p>2$的情况,我们的示例表明,尤其当使用子问题的全局极小值时,即使在渐近意义下,也并非所有迭代都能成功。只有当使用“正确的”局部模型极小值时,非自适应情形下的$p/(q-1)$阶局部收敛性在$p>q-1$时才能保持,否则超线性速率会下降。由此我们证实,只要$p$足够大,自适应高阶方法在某些退化问题上能实现超线性收敛,并给出了极限情况下可预期的收敛阶的精确界。
英文摘要:
Optimization methods that make use of derivatives of the objective function up to order $p > 2$ are called tensor methods. Among them, ones that minimize a regularized $p$th-order Taylor expansion at each step have been shown to possess optimal global complexity, which improves as $p$ increases. The local convergence of such optimization algorithms on functions that have Lipschitz continuous $p$th derivatives and are uniformly convex of order $q$ has been studied by Doikov and Nesterov [Math. Program., 193 (2022), pp. 315--336]. We extend these local convergence results to locally uniformly convex functions and fully adaptive methods, which do not need knowledge of the Lipschitz constant, thus providing the first sharp local rates for AR$p$. We discuss the surprising new challenges encountered by nonconvex local models and non-unique model minimizers. For $p > 2$, our examples show that in particular when using the global minimizer of the subproblem, even asymptotically not all iterations need to be successful. Only if the "right" local model minimizer is used, the $p/(q-1)$th-order local convergence from the non-adaptive case is preserved for $p > q-1$, otherwise the superlinear rate can degrade. We thus confirm that adaptive higher-order methods achieve superlinear convergence for certain degenerate problems as long as $p$ is large enough and provide sharp bounds on the order of convergence one can expect in the limit.
Alternating minimization algorithms for graph regularized tensor completion
Huawei European Research Institute(华为欧洲研究院);LISN, INRIA, Université Paris-Saclay(法国国家信息与自动化研究所,巴黎萨克雷大学);LSEC, Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院科学与工程计算国家重点实验室);Department of INMA, Université catholique de Louvain(鲁汶天主教大学)
A Flexible Optimization Framework for Regularized Matrix-Tensor Factorizations with Linear Couplings
Simula Metropolitan Center for Digital Engineering(Simula城市数字工程中心);Oslo Metropolitan University(奥斯陆城市大学);CNRS(法国国家科学研究中心);Université de Rennes(雷恩大学);Inria(法国国家信息与自动化研究所);IRISA(IRISA研究所)
Accelerating nuclear-norm regularized low-rank matrix optimization through Burer-Monteiro decomposition
Institute of Statistical Mathematics(统计数学研究所);Department of Mathematics University of Maryland at College Park(大学数学系);Institute of Operations Research and Analytics National University of Singapore(运营研究与分析国家大学研究所);Department of Mathematics and Institute of Operations Research and Analytics National University of Singapore(大学数学系和运营研究与分析国家大学研究所)