Darina Dvinskikh, Alexander Tyurin, Alexander Gasnikov, Sergey Omelchenko
In this paper, we propose a new way to obtain optimal convergence rates for smooth stochastic (strong) convex optimization tasks. Our approach is based on results for optimization tasks where gradients have nonrandom noise. In contrast to previously known results, we extend our idea to the inexact model conception.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
In this paper, we describe a new way to get convergence rates for optimal methods in smooth (strongly) convex optimization tasks. Our approach is based on results for tasks where gradients have nonrandom small noises. Unlike previous results, we obtain convergence rates with model conception.
We show how one can obtain nonaccelerated randomized coordinate descent method (Yu. Nesterov, 2010) and nonaccelerated method of randomization of sum-type functional (Le Roux-Schmidt-Bach, 2012) from the optimal method for the stochastic optimization problem (SIGMA, Devolder-Glineur-Nesterov-Dvurechensky-Gasnikov, 2014). The main trick is a special restart technique. We considered this trick to be usefull in others contexts. We consider only strongly convex case. We show that accelerated variants of this methods seems to be nontrivial in this context. That is, it is hard (perhaps impossible) t…
The $\ell_p$ regularization problem with $0< p< 1$ has been widely studied for finding sparse solutions of linear inverse problems and gained successful applications in various mathematics and applied science fields. The proximal gradient algorithm is one of the most popular algorithms for solving the $\ell_p$ regularisation problem. In the present paper, we investigate the linear convergence issue of one inexact descent method and two inexact proximal gradient algorithms (PGA). For this purpose, an optimality condition theorem is explored to provide the equivalences among a local minimum, sec…
Difference-of-Convex Optimization via Inexact Smoothing Descent Methods: Difference of High-Order Moreau Envelopes
提出HOME-DC函数并建立其性质，通过非精确一阶预言机设计下降方法求解DC优化问题，理论分析收敛性，稀疏聚类实验验证有效性。
We study the convergence issue for inexact descent algorithm (employing general step sizes) for multiobjective optimizations on general Riemannian manifolds (without curvature constraints). Under the assumption of the local convexity/quasi-convexity, local/global convergence results are established. On the other hand, without the assumption of the local convexity/quasi-convexity, but under a Kurdyka-Łojasiewicz-like condition, local/global linear convergence results are presented, which seem new even in Euclidean spaces setting and improve sharply the corresponding results in [24] in the case …
Lipschitz-Free Mirror Descent Methods for Relatively Strongly Convex Functions with/without Absolute and Relative Inexactness
本文提出了一种无需Lipschitz连续性假设的镜像下降方法，用于解决相对强凸函数的非光滑优化问题，并通过实验验证了其有效性。
ItsDEAL: Inexact two-level smoothing descent algorithms for weakly convex optimization
本文提出 ItsDEAL 框架，利用高阶 Moreau 包络对弱凸优化问题进行两层光滑化，并结合非精确一阶方法与无参数下降策略，在 KL 性质下证明了全局收敛性和线性速率，数值实验验证了其有效性。
Worst-case convergence analysis of relatively inexact gradient descent on smooth convex functions
本文利用PEP框架，对光滑凸函数上带相对误差界的梯度下降算法进行最坏情况收敛性分析，推导单步精确行为、多步上下界，并给出最优常数步长选择。
Inexact Riemannian Gradient Descent Method for Nonconvex Optimization
针对黎曼流形非凸优化中精确梯度难获取的问题，提出带收敛保证的新型非精确黎曼梯度下降算法，建立两个非精确梯度条件，兼具理论收敛性与实际应用效率。
An Inexact Riemannian Gradient Descent Algorithm on the Stiefel Manifold with One Newton-Schulz Iteration
本文提出一种Stiefel流形上带一次Newton-Schulz迭代的不精确黎曼梯度下降算法，可保证收敛性，性能优于同类算法，还提出其随机版本且表现优异。
Convergence of First-Order Algorithms with Momentum from the Perspective of an Inexact Gradient Descent Method
本文提出带动量不精确梯度下降法IGDm作为一阶动量方法的统一框架，在PLK条件下建立收敛速率，并推导EGm、SAMm和IPPm的全局与局部收敛性质，通过无导数优化实验验证动量效应。
We introduce an inexact variant of Stochastic Mirror Descent (SMD), called Inexact Stochastic Mirror Descent (ISMD), to solve nonlinear two-stage stochastic programs where the second stage problem has linear and nonlinear coupling constraints and a nonlinear objective function which depends on both first and second stage decisions. Given a candidate first stage solution and a realization of the second stage random vector, each iteration of ISMD combines a stochastic subgradient descent using a prox-mapping with the computation of approximate (instead of exact for SMD) primal and dual second st…
A New Inexact Gradient Descent Method with Applications to Nonsmooth Convex Optimization
本文提出一种新的非精确梯度法（IGD），在KL性质下保证收敛到稳定点，并将其应用于设计非精确邻近点法和非精确增广拉格朗日法，数值实验表明其优于现有非光滑凸优化算法。
Multi-agent systems are being increasingly deployed in challenging environments for performing complex tasks such as multi-target tracking, search-and-rescue, and intrusion detection. Notwithstanding the computational limitations of individual robots, such systems rely on collaboration to sense and react to the environment. This paper formulates the generic target tracking problem as a time-varying optimization problem and puts forth an inexact online gradient descent method for solving it sequentially. The performance of the proposed algorithm is studied by characterizing its dynamic regret, …
Block-coordinate descent (BCD) is a popular framework for large-scale regularized optimization problems with block-separable structure. Existing methods have several limitations. They often assume that subproblems can be solved exactly at each iteration, which in practical terms usually restricts the quadratic term in the subproblem to be diagonal, thus losing most of the benefits of higher-order derivative information. Moreover, in contrast to the smooth case, non-uniform sampling of the blocks has not yet been shown to improve the convergence rate bounds for regularized problems. This work p…
Riemannian Inexact Gradient Descent for Quadratic Discrimination
针对高维小样本成像中的二次判别任务，提出一种对不精确梯度鲁棒的黎曼优化算法，证明O(1/K)收敛率，并在格拉斯曼流形上验证性能。
We consider non-differentiable dynamic optimization problems such as those arising in robotics and subspace tracking. Given the computational constraints and the time-varying nature of the problem, a low-complexity algorithm is desirable, while the accuracy of the solution may only increase slowly over time. We put forth the proximal online gradient descent (OGD) algorithm for tracking the optimum of a composite objective function comprising of a differentiable loss function and a non-differentiable regularizer. An online learning framework is considered and the gradient of the loss function i…
In this paper, we develop a unified framework for analyzing the tracking error and dynamic regret of inexact online optimization methods under a variety of settings. Specifically, we leverage the quadratic constraint approach from control theory to formulate sequential semidefinite programs (SDPs) whose feasible points naturally correspond to tracking error bounds of various inexact online optimization methods including the inexact online gradient descent (OGD) method, the online gradient descent-ascent method, the online stochastic gradient method, and the inexact proximal online gradient met…
Inexact Online Proximal Mirror Descent for time-varying composite optimization
本文研究含梯度与近端算子误差的在线近端镜像下降算法，在凸光滑条件下获得了动态遗憾精确估计，消除了先前结果的$O(T)$项并将适用域扩展至全空间$\mathbb{R}^n$。
A Two-Dimensional Extremizer for Relatively Inexact Gradient Descent
本文构造了一个二维显式极值函数，证明了相对不精确梯度下降一步的最坏情况常数猜想，并指出二维在中间区域是必要的。
A Universal Transfer Theorem for Convex Optimization Algorithms Using Inexact First-order Oracles
本文提出了一种通用转换定理，能够将任何基于精确一阶信息的凸优化算法以黑盒方式转换为可处理非精确一阶信息的算法，适用于更广泛的算法类别及结构化非凸问题。
We show that the primal-dual gradient method, also known as the gradient descent ascent method, for solving convex-concave minimax problems can be viewed as an inexact gradient method applied to the primal problem. The gradient, whose exact computation relies on solving the inner maximization problem, is computed approximately by another gradient method. To model the approximate computational routine implemented by iterative algorithms, we introduce the notion of dynamic inexact oracles, which are discrete-time dynamical systems whose output asymptotically approaches the output of an exact ora…
An Inexact Riemannian Proximal Momentum Variance-Reduced Method: Complexity Bounds and KL Sequential Convergence
针对紧嵌入子流形上的有限和非光滑复合问题，提出不精确黎曼邻近动量方差缩减方法，建立其复杂度界，并基于普通逐点KL性质推导带记忆的条件期望下降抽象KL原理，证明序列收敛性与速率。
In this paper we consider the problem of minimizing a convex function using a randomized block coordinate descent method. One of the key steps at each iteration of the algorithm is determining the update to a block of variables. Existing algorithms assume that in order to compute the update, a particular subproblem is solved exactly. In his work we relax this requirement, and allow for the subproblem to be solved inexactly, leading to an inexact block coordinate descent method. Our approach incorporates the best known results for exact updates as a special case. Moreover, these theoretical gua…
In this book we collect many different and useful facts around gradient descent method. First of all we consider gradient descent with inexact oracle. We build a general model of optimized function that include composite optimization approach, level's methods, proximal methods etc. Then we investigate primal-dual properties of the gradient descent in general model set-up. At the end we generalize method to universal one.
The self regulation problem as an inexact steepest descent method for multicriteria optimization
本文提出一种带Armijo规则的不精确最速下降法求解多准则优化问题，在拟凸性假设下证明序列收敛到临界Pareto点，并将其应用于心理学中的自调节模型。
An Inexact General Descent Method with Applications in Differential Equation-Constrained Optimization
针对微分方程约束优化中梯度依赖迭代求解、评估非精确的问题，提出非精确一般下降框架，证明两类步长下的全局收敛性，并通过自适应离散伴随梯度实验验证其效率。
Accelerated coordinate descent is a widely popular optimization algorithm due to its efficiency on large-dimensional problems. It achieves state-of-the-art complexity on an important class of empirical risk minimization problems. In this paper we design and analyze an accelerated coordinate descent (ACD) method which in each iteration updates a random subset of coordinates according to an arbitrary but fixed probability law, which is a parameter of the method. If all coordinates are updated in each iteration, our method reduces to the classical accelerated gradient descent method AGD of Nester…
We consider least squares semidefinite programming (LSSDP) where the primal matrix variable must satisfy given linear equality and inequality constraints, and must also lie in the intersection of the cone of symmetric positive semidefinite matrices and a simple polyhedral set. We propose an inexact accelerated block coordinate descent (ABCD) method for solving LSSDP via its dual, which can be reformulated as a convex composite minimization problem whose objective is the sum of a coupled quadratic function involving four blocks of variables and two separable non-smooth functions involving only …
An abstract convergence theorem for a class of generalized descent methods that explicitly models relative errors is proved. The convergence theorem generalizes and unifies several recent abstract convergence theorems. It is applicable to possibly non-smooth and non-convex lower semi-continuous functions that satisfy the Kurdyka--Lojasiewicz (KL) inequality, which comprises a huge class of problems. Most of the recent algorithms that explicitly prove convergence using the KL inequality can cast into the abstract framework in this paper and, therefore, the generated sequence converges to a stat…
On the Convergence of Inexact Gradient Descent with Controlled Synchronization Steps
针对对等网络下目标函数和最小化问题，提出含两阶段协调的类梯度算法，允许有界非精确梯度且无需梯度有界假设，通过本地可验证准则动态触发同步以降低通信开销，并证明了算法收敛性。
In this paper, we use Proximal Cubic regularized Newton Methods (PCNM) to optimize the sum of a smooth convex function and a non-smooth convex function, where we use inexact gradient and Hessian, and an inexact subsolver for the cubic regularized second-order subproblem. We propose inexact variants of PCNM and accelerated PCNM respectively, and show that both variants can achieve the same convergence rate as in the exact case, provided that the errors in the inexact gradient, Hessian and subsolver decrease at appropriate rates. Meanwhile, in the online stochastic setting where data comes endle…
Inexact proximal methods for weakly convex functions
本文提出不精确近端方法求解光滑与弱凸非光滑函数之和的稳定点，基于连续映射求零点框架，证明了近端点法与近端梯度法在KL性质下的全局收敛性与构造性收敛速率。
Normalized First-Order Methods for Convex (L0, L1)-Smooth Optimization with Inexact Gradients
针对需不精确梯度的凸(L0, L1)-光滑优化问题，提出两种基于比较预言机的归一化一阶方法，推导收敛速率并通过实验验证理论结果。
Convergence and Trade-Offs in Riemannian Gradient Descent and Riemannian Proximal Point
本文分析黎曼梯度下降与（不精确）黎曼近端点法的收敛速率，证明迭代自然保持在优化器周围球内，并提供可实现的算法及新性质，辅以实证验证。
Recently several methods were proposed for sparse optimization which make careful use of second-order information [10, 28, 16, 3] to improve local convergence rates. These methods construct a composite quadratic approximation using Hessian information, optimize this approximation using a first-order method, such as coordinate descent and employ a line search to ensure sufficient descent. Here we propose a general framework, which includes slightly modified versions of existing algorithms and also a new algorithm, which uses limited memory BFGS Hessian approximations, and provide a novel global…
Convergence Analysis of an Inexact Proximal Trust-Region Method
本文提出并分析了一种非精确近端信赖域算法，用于求解光滑非凸与非光滑凸函数之和的最小化问题，通过允许近端算子非精确评估保证全局收敛，并在PDE约束优化中验证了其有效性。
We provide new tools for worst-case performance analysis of the gradient (or steepest descent) method of Cauchy for smooth strongly convex functions, and Newton's method for self-concordant functions, including the case of inexact search directions. The analysis uses semidefinite programming performance estimation, as pioneered by Drori and Teboulle [Mathematical Programming, 145(1-2):451-482, 2014], and extends recent performance estimation results for the method of Cauchy by the authors [Optimization Letters, 11(7), 1185-1199, 2017]. To illustrate the applicability of the tools, we demonstra…
In this paper, we propose an inexact block coordinate descent algorithm for large-scale nonsmooth nonconvex optimization problems. At each iteration, a particular block variable is selected and updated by inexactly solving the original optimization problem with respect to that block variable. More precisely, a local approximation of the original optimization problem is solved. The proposed algorithm has several attractive features, namely, i) high flexibility, as the approximation function only needs to be strictly convex and it does not have to be a global upper bound of the original function…
Based on a result by Taylor, Hendrickx, and Glineur (J. Optim. Theory Appl., 178(2):455--476, 2018) on the attainable convergence rate of gradient descent for smooth and strongly convex functions in terms of function values, an elementary convergence analysis for general descent methods with fixed step sizes is presented. It covers general variable metric methods, gradient related search directions under angle and scaling conditions, as well as inexact gradient methods. In all cases, optimal rates are obtained.
Variable order structures model situations in which the comparison between two points depends on a point-to-cone map. In this paper, an inexact projected gradient method for solving smooth constrained vector optimization problems on variable ordered spaces is presented. It is shown that every accumulation point of the generated sequence satisfies the first order necessary optimality condition. The convergence of all accumulation points to a weakly efficient point is established under suitable convexity assumptions for the objective function. The convergence results are also derived in the part…
In this paper, elliptic control problems with integral constraint on the gradient of the state and box constraints on the control are considered. The optimal conditions of the problem are proved. To numerically solve the problem, we use the 'First discretize, then optimize' approach. Specifically, we discretize both the state and the control by piecewise linear functions. To solve the discretized problem efficiently, we first transform it into a multi-block unconstrained convex optimization problem via its dual, then we extend the inexact majorized accelerating block coordinate descent (imABCD…
In this paper we propose and analyze two dual methods based on inexact gradient information and averaging that generate approximate primal solutions for smooth convex optimization problems. The complicating constraints are moved into the cost using the Lagrange multipliers. The dual problem is solved by inexact first order methods based on approximate gradients and we prove sublinear rate of convergence for these methods. In particular, we provide, for the first time, estimates on the primal feasibility violation and primal and dual suboptimality of the generated approximate primal and dual so…
Accelerated iterative regularization via dual diagonal descent
提出加速迭代双重对角下降算法求解线性逆问题，分析惯性方法的收敛性与稳定性，并利用非精确近端演算证明早停最优收敛速率。
In this paper we propose accelerated gradient descent schemes for convex optimization problems in Hilbert space. We consider inexact oracle case.
Centered Weak Discrete Riemannian Gradients: A Unified Framework for Riemannian Optimization
提出中心弱离散黎曼梯度（c-wDRG）框架，统一分析黎曼流形优化方法，推导收敛结果并开发加速方案，为相关方法提供显式步长条件与收敛速率。
An Inexact Augmented Lagrangian Method for $(L_0, L_1)$-Smooth Convex Optimization
本文针对带$(L_0, L_1)$-光滑目标函数的线性约束凸优化问题，提出一种不精确增广拉格朗日方法，结合两阶段加速过程求解，突破了经典方法的梯度利普希茨连续性假设限制。
We introduce a class of specially structured linear programming (LP) problems, which has favorable modeling capability for important application problems in different areas such as optimal transport, discrete tomography and economics. To solve these generally large-scale LP problems efficiently, we design an implementable inexact entropic proximal point algorithm (iEPPA) combined with an easy-to-implement dual block coordinate descent method as a subsolver. Unlike existing entropy-type proximal point algorithms, our iEPPA employs a more practically checkable stopping condition for solving the …
Accelerating Inexact HyperGradient Descent for Bilevel Optimization
提出RAHGD方法求解非凸-强凸双层优化问题，以最优复杂度找到一阶和二阶驻点，刷新了该领域及极小极大优化的理论基准。
In this paper we present a convergence rate analysis of inexact variants of several randomized iterative methods. Among the methods studied are: stochastic gradient descent, stochastic Newton, stochastic proximal point and stochastic subspace ascent. A common feature of these methods is that in their update rule a certain sub-problem needs to be solved exactly. We relax this requirement by allowing for the sub-problem to be solved inexactly. In particular, we propose and analyze inexact randomized iterative methods for solving three closely related problems: a convex stochastic quadratic optim…
The state-of-the-art methods for solving optimization problems in big dimensions are variants of randomized coordinate descent (RCD). In this paper we introduce a fundamentally new type of acceleration strategy for RCD based on the augmentation of the set of coordinate directions by a few spectral or conjugate directions. As we increase the number of extra directions to be sampled from, the rate of the method improves, and interpolates between the linear rate of RCD and a linear rate independent of the condition number. We develop and analyze also inexact variants of these methods where the sp…
Inexact Adjoint Gradients and Directional Tolerances for Full-Potential Airfoil Optimization
本文提出一种将离散伴随梯度误差分析与使用方向误差容差的优化方法相结合的框架，并将其应用于由保守全势流求解器在贴体结构网格上控制的空气foil形状优化问题。理论部分推导了标量目标函数的简化离散伴随公式，并分析了不精确状态和伴随残差如何传播到简化梯度中。对于状态变量为仿射的残差，梯度误差被界于状态和伴随残差容差的线性组合。在决策变量的紧集上，获得了一个统一版本的此界，从而得到一个方向容差条件，使得不精确梯度满足精确下降不等式。所得到的不精确通用方向法在均匀有界、递减和Armijo型步长规则下继承了收敛性质。计算部分结合了抛物线初始网格生成器、椭圆网格平滑器和全势离散化与人工密度稳定化和近似因子化迭代。将优化问题表述为压力匹配问题，其中一类形状变换空气foil参数化被调整，使计算得到的表面压力系数接近给定的参考数据，受网格生成和全势残差约束。
Primal-Dual Inexact Newton-MR for Nonconvex Optimization with Equality Constraints
针对带等式约束的非凸优化，提出一种基于共轭残差法的非精确原始-对偶牛顿方法，直接处理子问题非正定性，避免有害正则化，并具有全局收敛保证和良好实证表现。
First-order majorization-minimization meets high-order majorant: Boosted inexact high-order forward-backward method
本文提出基于高阶majorant的一阶majorization-minimization框架，开发了高阶非精确前向-后向算法（HiFBA）及其增强变体，在Kurdyka-Łojasiewicz条件下证明全局线性收敛，并在线性逆问题和NMF实验中展示高效性。
In this paper, we present a convergence rate analysis for the inexact Krasnosel'skii-Mann iteration built from nonexpansive operators. Our results include two main parts: we first establish global pointwise and ergodic iteration-complexity bounds, and then, under a metric subregularity assumption, we establish local linear convergence for the distance of the iterates to the set of fixed points. The obtained iteration-complexity result can be applied to analyze the convergence rate of various monotone operator splitting methods in the literature, including the Forward-Backward, the Generalized …
Federated Learning via Inexact ADMM
本文针对联邦学习优化效率问题，提出一种非精确ADMM算法，在温和条件下实现收敛，兼具计算与通信效率并能对抗滞后效应，数值性能优于现有主流联邦学习算法。
Mirror descent actor-critic methods for entropy regularised MDPs in general spaces: stability and convergence
本文提出了一种用于熵正则化MDP的镜像下降Actor-Critic方法，分析了其在一般空间中的稳定性与收敛性，证明了在不同TD步骤数增长条件下算法的亚线性或线性收敛性。
An Inexact Variable Metric Proximal Gradient-subgradient Algorithm for a Class of Fractional Optimization Problems
研究一类分数优化问题，提出不精确变度量近端梯度 - 次梯度算法（iVPGSA），它能灵活近似求解子问题，适应性强且效率高。基于KL性质证明其全局收敛并刻画速率，数值实验显示该算法优于现有算法。
Convergence of Descent Optimization Algorithms under Polyak-Łojasiewicz-Kurdyka Conditions
本文针对非光滑非凸优化中一般类别的下降算法，在多种PLK型条件下开展全面收敛性分析，证明了较低指数PLK条件下算法的有限终止性，建立了非精确约化梯度等方法的新收敛速率，并揭示了PLK条件与梯度连续性的兼容性规律。
Decentralized Constraint-Coupled Optimization with Inexact Oracle
针对带全局耦合等式约束的去中心化优化问题，提出采用可控不精确度非精确对偶梯度的iDDGT方法，计算效率更优且可在无约束矩阵条件的有向图上线性收敛，适用范围更广。
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。