Flavien Léger
We present a new perspective on the popular Sinkhorn algorithm, showing that it can be seen as a Bregman gradient descent (mirror descent) of a relative entropy (Kullback-Leibler divergence). This viewpoint implies a new sublinear convergence rate with a robust constant.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
We present a new perspective on the celebrated Sinkhorn algorithm by showing that is a special case of incremental/stochastic mirror descent. In order to see this, one should simply plug Kullback-Leibler divergence in both mirror map and the objective function. Since the problem has unbounded domain, the objective function is neither smooth nor it has bounded gradients. However, one can still approach the problem using the notion of relative smoothness, obtaining that the stochastic objective is 1-relative smooth. The discovered equivalence allows us to propose 1) new methods for optimal trans…
We analyze two algorithms for approximating the general optimal transport (OT) distance between two discrete distributions of size $n$, up to accuracy $\varepsilon$. For the first algorithm, which is based on the celebrated Sinkhorn's algorithm, we prove the complexity bound $\widetilde{O}\left({n^2/\varepsilon^2}\right)$ arithmetic operations. For the second one, which is based on our novel Adaptive Primal-Dual Accelerated Gradient Descent (APDAGD) algorithm, we prove the complexity bound $\widetilde{O}\left(\min\left\{n^{9/4}/\varepsilon, n^{2}/\varepsilon^2 \right\}\right)$ arithmetic opera…
Many problems in machine learning can be formulated as optimizing a convex functional over a vector space of measures. This paper studies the convergence of the mirror descent algorithm in this infinite-dimensional setting. Defining Bregman divergences through directional derivatives, we derive the convergence of the scheme for relatively smooth and convex pairs of functionals. Such assumptions allow to handle non-smooth functionals such as the Kullback--Leibler (KL) divergence. Applying our result to joint distributions and KL, we show that Sinkhorn's primal iterations for entropic optimal tr…
A Bregman-Sinkhorn Algorithm for the Maximum Weight Independent Set Problem
本文提出一种基于对偶坐标下降与熵平滑调度的Bregman-Sinkhorn算法，用于求解NP难的最大权独立集问题，可在数秒内为含88.2万节点、3.44亿边的真实图生成高质量近似解。
The Birkhoff Geometry of Manifold-Constrained Hyper-Connections: Two Channels, Vertex Viscosity, and Sinkhorn as a Retraction
本文从Birkhoff几何角度分析流形约束超连接（mHC），揭示其双通道动力学、Sinkhorn作为回缩的几何性质，并证明梯度流与镜像下降的收敛速率差异。
Annealed Sinkhorn for Optimal Transport: convergence, regularization path and debiasing
提出Debiased Annealed Sinkhorn算法，通过减少松弛误差实现更快的退火调度，单次运行即可覆盖标准Sinkhorn算法的完整速度-精度Pareto前沿。
Nested Stochastic Algorithm for Generalized Sinkhorn distance-Regularized Distributionally Robust Optimization
本文针对广义Sinkhorn距离正则化的非凸分布鲁棒优化问题，推导出嵌套随机优化的对偶形式，提出嵌套SGD算法并证明其多项式复杂度，数值实验验证了算法的效率与鲁棒性。
Motivated by gradient methods in optimization theory, we give methods based on $ψ$-fractional derivatives of order $α$ in order to solve unconstrained optimization problems. The convergence of these methods is analyzed in detail. This paper also presents an Adams-Bashforth-Moulton (ABM) method for the estimation of solutions to equations involving $ψ$-fractional derivatives. Numerical examples using the ABM method show that the fractional order $α$ and weight $ψ$ are tunable parameters, which can be helpful for improving the performance of gradient descent methods.
On Partial Optimal Transport: Revising the Infeasibility of Sinkhorn and Efficient Gradient Methods
本文针对部分最优传输问题，指出Sinkhorn算法因取整不兼容而不可行，提出新取整算法及修正Sinkhorn，并开发两种高效一阶方法，在复杂度上达到最优。
On Sinkhorn's Algorithm and Choice Modeling
本文证明基于 Luce 选择公理的一类选择模型的最大似然估计等价于矩阵平衡问题，从而统一现有算法为 Sinkhorn 算法实例，并首次给出一般非负矩阵下 Sinkhorn 算法的全局线性收敛性及精确渐近速率。
Convergence of First-Order Algorithms with Momentum from the Perspective of an Inexact Gradient Descent Method
本文提出带动量不精确梯度下降法IGDm作为一阶动量方法的统一框架，在PLK条件下建立收敛速率，并推导EGm、SAMm和IPPm的全局与局部收敛性质，通过无导数优化实验验证动量效应。
A Mirror Descent Perspective of Smoothed Sign Descent
本文从镜像下降视角研究平滑符号下降在回归问题中的动力学，提出镜像映射建立对偶等价性，并将收敛解刻画为最小化Bregman散度型函数的近似KKT点，揭示了调节稳定性常数可降低KKT误差。
The starting assumptions to study the convergence and complexity of gradient-type methods may be the smoothness (also called Lipschitz continuity of gradient) and the strong convexity. In this note, we revisit these two basic properties from a new perspective that motivates their definitions and equivalent characterizations, along with an improved linear convergence of the gradient descent method.
First-order Methods for Unconstrained Vector Optimization Problems: A Unified Majorization-Minimization Perspective
本文提出了一种统一的主导最小化方法，通过缩小替代函数与目标函数之间的差距来提升一阶方法在无约束向量优化问题中的性能。
Solving systems of linear equations is one of the most important primitives in quantum computing that has the potential to provide a practical quantum advantage in many different areas, including in optimization, simulation, and machine learning. In this work, we revisit quantum linear system solvers from the perspective of convex optimization, and in particular gradient descent-type algorithms. This leads to a considerable constant-factor improvement in the runtime (or, conversely, a several orders of magnitude smaller error with the same runtime/circuit depth). More precisely, we first show …
Frictionless Hamiltonian Descent and Coordinate Hamiltonian Descent for Strongly Convex Quadratic Problems
本文提出无摩擦哈密顿下降法及坐标哈密顿下降法，用于解决强凸二次问题，展示了其在加速速率和线性方程组求解中的广泛应用。
The Anytime Convergence of Stochastic Gradient Descent with Momentum: From a Continuous-Time Perspective
本文从连续时间视角分析SGDM，证明其轨迹收敛于确定性二阶ODE，并利用Lyapunov函数推导出算法在整个轨迹上的随时收敛保证。
A Unifying Perspective on Descent Directions for Constrained Minimization and Switching Criteria
本文提出Shadow-CG方法，利用负梯度投影的阴影方向作为最优下降方向，结合Frank-Wolfe顶点步，实现线性收敛，其速率依赖于投影曲线断点数而非金字塔宽度，并给出断点数量的界。
This paper considers the analysis of continuous time gradient-based optimization algorithms through the lens of nonlinear contraction theory. It demonstrates that in the case of a time-invariant objective, most elementary results on gradient descent based on convexity can be replaced by much more general results based on contraction. In particular, gradient descent converges to a unique equilibrium if its dynamics are contracting in any metric, with convexity of the cost corresponding to the special case of contraction in the identity metric. More broadly, contraction analysis provides new ins…
Convergence of two-timescale gradient descent ascent dynamics: finite-dimensional and mean-field perspectives
本文分析双时间尺度梯度下降上升算法在有限维和平均场设置下的收敛性，通过次相干性方法和混合同步-反射耦合技术，分别证明了近准静态区域和有限尺度比值下的收敛结果。
Reinterpreting EMML as Mirror Descent for Constrained Maximum Likelihood Estimation
本文将EMML算法重新解释为镜像下降法，通过重新参数化的目标函数引入凸约束，保持EMML更新的乘法结构以提高效率，并证明算法收敛性，实验表明其在超光谱解混中收敛更快。
Designing Algorithms for Entropic Optimal Transport from an Optimisation Perspective
针对熵正则化最优传输问题，从优化视角提出基于半对偶和非凸约束的新方法，给出非渐近收敛速率及动量加速保证，并将框架推广至动态 Schrödinger 桥问题。
Research on the descent direction of prediction correction algorithms for pseudo-convex/convex optimization problems
本文研究了伪凸和凸优化问题中预测修正算法的下降方向调整系数，通过微分方程视角分析其范围，并验证了算法的收敛性及有效性。
Descent Properties of an Anderson Accelerated Gradient Method With Restarting
本文研究带重启的 Anderson 加速梯度方法的局部下降性质，证明其能比普通梯度法更快降低目标函数值，为 AA 在非凸优化中的收敛分析提供新视角。
UGM: A Unified Framework and New Perspectives for Accelerated Gradient Methods in Smooth and Strongly Convex Optimization
本文提出名为UGM的统一框架，将重球法与梯度下降结合，开发增强型加速梯度算法，经实验验证其在两类优化问题上性能优于基线方法。
A New Perspective of Accelerated Gradient Methods: The Controlled Invariant Manifold Approach
本文从受控动力系统视角提出一种加速梯度算法生成方法，旨在以最少迭代次数到达满足梯度为零的最优解，从而降低计算复杂度。
Computing Stationary Distribution via Dirichlet-Energy Minimization by Coordinate Descent
本文提出基于优化的RLGL算法，用于大马尔可夫链平稳分布的计算，并通过坐标下降法实现指数收敛性。
Finding the mixed Nash equilibria (MNE) of a two-player zero sum continuous game is an important and challenging problem in machine learning. A canonical algorithm to finding the MNE is the noisy gradient descent ascent method which in the infinite particle limit gives rise to the {\em Mean-Field Gradient Descent Ascent} (GDA) dynamics on the space of probability measures. In this paper, we first study the convergence of a two-scale Mean-Field GDA dynamics for finding the MNE of the entropy-regularized objective. More precisely we show that for each finite temperature (or regularization parame…
Regularization properties of dual subgradient flow
本文研究对偶次梯度流的正则化性质，证明其在处理仅凸正则项时能达到与惩罚方法相同的恢复精度，且计算效率更高。
Optimized methods for composite optimization: a reduction perspective
本文提出一种通用框架，通过代数恒等式将无约束平滑优化的优化方法直接归约推导至复合优化设定，并据此证明了近端梯度下降的步长加速现象、比 FISTA 更快的收敛速率及最小化梯度范数的新方法。
In this manuscript, we analyze the sparse signal recovery (compressive sensing) problem from the perspective of convex optimization by stochastic proximal gradient descent. This view allows us to significantly simplify the recovery analysis of compressive sensing. More importantly, it leads to an efficient optimization algorithm for solving the regularized optimization problem related to the sparse recovery problem. Compared to the existing approaches, there are two advantages of the proposed algorithm. First, it enjoys a geometric convergence rate and therefore is computationally efficient. S…
New Perspectives on the Polyak Stepsize: Surrogate Functions and Negative Results
本文从替代损失的角度重新审视Polyak步长及其变种，揭示其收敛性质，并证明某些上界中的非收敛结果确实存在。
Mini-Batch Covariance, Diffusion Limits, and Oracle Complexity in Stochastic Gradient Descent: A Sampling-Design Perspective
本文从采样设计角度研究了随机梯度下降中最小批量梯度噪声的协方差特性，推导了扩散极限与Oracle复杂度的理论边界，并通过数值实验验证了相关结论。
Optimisers are an essential component for training machine learning models, and their design influences learning speed and generalisation. Several studies have attempted to learn more effective gradient-descent optimisers via solving a bi-level optimisation problem where generalisation error is minimised with respect to optimiser parameters. However, most existing optimiser learning methods are intuitively motivated, without clear theoretical support. We take a different perspective starting from mirror descent rather than gradient descent, and meta-learning the corresponding Bregman divergenc…
Zeroth-order optimization is the process of minimizing an objective $f(x)$, given oracle access to evaluations at adaptively chosen inputs $x$. In this paper, we present two simple yet powerful GradientLess Descent (GLD) algorithms that do not rely on an underlying gradient estimate and are numerically stable. We analyze our algorithm from a novel geometric perspective and present a novel analysis that shows convergence within an $ε$-ball of the optimum in $O(kQ\log(n)\log(R/ε))$ evaluations, for any monotone transform of a smooth and strongly convex objective with latent dimension $k < n$, wh…
We propose a novel actor-critic algorithm with guaranteed convergence to an optimal policy for a discounted reward Markov decision process. The actor incorporates a descent direction that is motivated by the solution of a certain non-linear optimization problem. We also discuss an extension to incorporate function approximation and demonstrate the practicality of our algorithms on a network routing application.
A Large Deviations Perspective on Policy Gradient Algorithms
本文从大偏差理论视角，为非凸目标下随机梯度下降的迭代序列识别出速率函数，并利用收缩原理将特定策略梯度的收敛性质推广至更广泛的参数化。
Many machine learning and data science tasks require solving non-convex optimization problems. When the loss function is a sum of multiple terms, a popular method is the stochastic gradient descent. Viewed as a process for sampling the loss function landscape, the stochastic gradient descent is known to prefer flat minima. Though this is desired for certain optimization problems such as in deep learning, it causes issues when the goal is to find the global minimum, especially if the global minimum resides in a sharp valley. Illustrated with a simple motivating example, we show that the fundame…
Target Mirror Descent: A Unifying Framework for Solving Monotone Variational Inequalities
本文提出目标镜像下降框架，通过双变量修正机制稳定单调流，统一了近端点算法、extragradient方法等经典算法，并改进了折扣镜像下降的均衡偏差。
Adaptive control mechanisms in gradient descent algorithms
针对凸且局部光滑函数的梯度下降法，从自适应控制视角设计结合过去测量与未来预测信息的步长更新规则，通过李雅普诺夫分析平衡收敛速率与鲁棒性，为新型自适应优化算法设计提供了有前景的方法。
Gradient Flow Sampler-based Distributionally Robust Optimization
提出一种基于PDE梯度流框架的分布鲁棒优化方法，利用MCMC采样与梯度流理论实现最坏情况分布采样，并统一了现有DRO方法。
Nesterov's accelerated gradient descent (AGD), an instance of the general family of "momentum methods", provably achieves faster convergence rate than gradient descent (GD) in the convex setting. However, whether these methods are superior to GD in the nonconvex setting remains open. This paper studies a simple variant of AGD, and shows that it escapes saddle points and finds a second-order stationary point in $\tilde{O}(1/ε^{7/4})$ iterations, faster than the $\tilde{O}(1/ε^{2})$ iterations required by GD. To the best of our knowledge, this is the first Hessian-free algorithm to find a second…
In this paper, we introduce the tamed stochastic gradient descent method (TSGD) for optimization problems. Inspired by the tamed Euler scheme, which is a commonly used method within the context of stochastic differential equations, TSGD is an explicit scheme that exhibits stability properties similar to those of implicit schemes. As its computational cost is essentially equivalent to that of the well-known stochastic gradient descent method (SGD), it constitutes a very competitive alternative to such methods. We rigorously prove (optimal) sub-linear convergence of the scheme for strongly conve…
In distributed machine learning, efficient training across multiple agents with different data distributions poses significant challenges. Even with a centralized coordinator, current algorithms that achieve optimal communication complexity typically require either large minibatches or compromise on gradient complexity. In this work, we tackle both centralized and decentralized settings across strongly convex, convex, and nonconvex objectives. We first demonstrate that a basic primal-dual method, (Accelerated) Gradient Ascent Multiple Stochastic Gradient Descent (GA-MSGD), applied to the Lagra…
Input-to-State Stability of a Bilevel Proximal Gradient Descent Algorithm
本文针对控制感知设计优化中的双层优化问题，提出了一种互连的近端梯度算法，并利用输入到状态稳定性理论从控制角度分析了其鲁棒性，推导出确保该算法收敛到最优解的条件。
The Limit Points of (Optimistic) Gradient Descent in Min-Max Optimization
本文从动力系统视角刻画了GDA和OGDA在最小-最大优化中的极限点，证明二者几乎对所有初始化避开不稳定临界点，并揭示了OGDA稳定点集包含GDA稳定点集且均包含局部最小-最大解的关系。
We take a Hamiltonian-based perspective to generalize Nesterov's accelerated gradient descent and Polyak's heavy ball method to a broad class of momentum methods in the setting of (possibly) constrained minimization in Euclidean and non-Euclidean normed vector spaces. Our perspective leads to a generic and unifying nonasymptotic analysis of convergence of these methods in both the function value (in the setting of convex optimization) and in norm of the gradient (in the setting of unconstrained, possibly nonconvex, optimization). Our approach relies upon a time-varying Hamiltonian that produce…
Stochastic Semi-Gradient Descent for Learning Mean Field Games with Population-Aware Function Approximation
针对传统平均场博弈（MFGs）学习方法的效率与稳定性问题，提出将策略和群体视为统一参数的视角，开发SemiSGD方法及群体感知线性函数近似（PA-LFA），并通过理论分析与实验验证其收敛性与有效性。
In this paper we analyze boosting algorithms in linear regression from a new perspective: that of modern first-order methods in convex optimization. We show that classic boosting algorithms in linear regression, namely the incremental forward stagewise algorithm (FS$_\varepsilon$) and least squares boosting (LS-Boost($\varepsilon$)), can be viewed as subgradient descent to minimize the loss function defined as the maximum absolute correlation between the features and residuals. We also propose a modification of FS$_\varepsilon$ that yields an algorithm for the Lasso, and that may be easily ext…
A Bregman Proximal Perspective on Classical and Quantum Blahut-Arimoto Algorithms
本文从 Bregman 近端视角统一解释经典与量子 Blahut-Arimoto 算法，证明其为镜像下降的特殊实例，并利用相对光滑性和强凸性恢复收敛速率，进而将该框架推广到能量约束信道容量、率失真函数及纠缠相对熵近似等问题。
Gradient Descent as a Perceptron Algorithm: Understanding Dynamics and Implicit Acceleration
本文研究了梯度下降在神经网络训练中的优化动态和隐式加速现象，通过非线性模型分析显示梯度下降步骤等价于广义感知机算法，揭示了非线性模型在迭代复杂度上的优势。
Randomised Splitting Methods and Stochastic Gradient Descent
本文揭示了随机梯度下降与常微分方程分裂方法的联系，提出了一种结合动量的对称小批量策略，显著降低了随机梯度偏差，并利用 Lyapunov 技术证明了其在强凸光滑函数上的收敛性及递减步长下的更快收敛率。
Natural gradient descent with momentum
本文提出一种自然梯度下降的动量版本，用于改进非线性模型类的学习过程，通过结合惯性动态方法如Heavy-Ball或Nesterov，提升优化性能。
Gradient is All You Need? How Consensus-Based Optimization can be Interpreted as a Stochastic Relaxation of Gradient Descent
本文将基于共识的优化解释为梯度下降的随机松弛，揭示其在非凸优化中的全局收敛性及对能量壁垒的克服能力。
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。