首页/论文/1909.03371

arXiv 1909.033712019-09-08math.OCstat.ML

On the connections between algorithmic regularization and penalization for convex losses

Qian Qian, Xiaoyuan Qian

英文摘要：

In this work we establish the equivalence of algorithmic regularization and explicit convex penalization for generic convex losses. We introduce a geometric condition for the optimization path of a convex function, and show that if such a condition is satisfied, the optimization path of an iterative algorithm on the unregularized optimization problem can be represented as the solution path of a corresponding penalized problem.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Solution of Optimal Power Flow Problems using Moment Relaxations Augmented with Objective Function Penalization

The optimal power flow (OPF) problem minimizes the operating cost of an electric power system. Applications of convex relaxation techniques to the non-convex OPF problem have been of recent interest, including work using the Lasserre hierarchy of "moment" relaxations to globally solve many OPF problems. By preprocessing the network model to eliminate low-impedance lines, this paper demonstrates the capability of the moment relaxations to globally solve large OPF problems that minimize active power losses for portions of several European power systems. Large problems with more general objective…

2015-08-20math.OC

通过算法等价性在隐凸损失上的在线学习：最优遗憾、几何障碍与Bandit反馈

Online Learning on Hidden-Convex Losses via Algorithmic Equivalence: Optimal Regret, Geometric Barrier, and Bandit Feedback

Singapore University of Technology and Design（新加坡科技设计大学）；National Technical University of Athens（雅典国家技术大学）；National and Kapodistrian University of Athens（雅典国家与卡多斯大学）；University of California, Irvine（加州大学 Irvine 分校）；Archimedes, Athena Research Center, Greece（希腊阿提卡研究中心 Archimedes）；National University of Singapore, Centre for Quantum Technologies（新加坡国立大学量子技术中心）

本文通过更精确的离散时间算法等价性论证，证明在线梯度下降在隐凸损失上达到最优的$\mathcal{O}(\sqrt{T})$遗憾，并澄清了所需几何条件，同时扩展到单点Bandit反馈得到$\mathcal{O}(T^{3/4})$期望遗憾。

2026-05-25cs.LGmath.OCstat.ML

A Laplacian-Based Approach for Finding Near Globally Optimal Solutions to OPF Problems

A semidefinite programming (SDP) relaxation globally solves many optimal power flow (OPF) problems. For other OPF problems where the SDP relaxation only provides a lower bound on the objective value rather than the globally optimal decision variables, recent literature has proposed a penalization approach to find feasible points that are often nearly globally optimal. A disadvantage of this penalization approach is the need to specify penalty parameters. This paper presents an alternative approach that algorithmically determines a penalization appropriate for many OPF problems. The proposed ap…

2016-04-02math.OC

经验X-risk最小化的算法基础

Algorithmic Foundations of Empirical X-risk Minimization

Texas A&M University（德克萨斯A&M大学）

本文提出经验X-risk最小化（EXM）框架，将AUROC、NDCG等非可分解目标统一建模为三类非凸优化问题，并给出基线算法与复杂度分析，为深度学习中的组合性风险优化奠定算法基础。

2023-10-27cs.AIcs.LGmath.OC

The importance of better models in stochastic optimization

Standard stochastic optimization methods are brittle, sensitive to stepsize choices and other algorithmic parameters, and they exhibit instability outside of well-behaved families of objectives. To address these challenges, we investigate models for stochastic minimization and learning problems that exhibit better robustness to problem families and algorithmic parameters. With appropriately accurate models---which we call the aProx family---stochastic methods can be made stable, provably convergent and asymptotically optimal; even modeling that the objective is nonnegative is sufficient for th…

2019-03-20math.OCstat.ML

Stability of Stochastic Gradient Descent on Nonsmooth Convex Losses

The Ohio State University（俄亥俄州立大学）；Google Research（谷歌研究院）；Pontificia Universidad Católica de Chile（智利天主教大学）

Uniform stability is a notion of algorithmic stability that bounds the worst case change in the model output by the algorithm when a single data point in the dataset is replaced. An influential work of Hardt et al. (2016) provides strong upper bounds on the uniform stability of the stochastic gradient descent (SGD) algorithm on sufficiently smooth convex losses. These results led to important progress in understanding of the generalization properties of SGD and several applications to differentially private convex optimization for smooth losses. Our work is the first to address uniform stabili…

2020-06-12cs.LGmath.OCstat.ML

更多相关论文收起更多论文

（含噪）随机梯度下降的时间一致 Wasserstein 稳定性界

Uniform-in-Time Wasserstein Stability Bounds for (Noisy) Stochastic Gradient Descent

Florida State University（佛罗里达州立大学）；Rutgers Business School（罗格斯商学院）；Princeton University（普林斯顿大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）

本文通过学习理论与应用概率的联系，提出统一的 Wasserstein 稳定性界证明准则，并基于 Lyapunov 函数为 SGD 在强凸、非凸及无额外噪声的凸与非凸损失下建立了时间一致稳定性界。

2023-10-28cs.LGmath.OCstat.ML

更好的压缩驱动器？考虑粘热损失的 CutFEM 3D 形状优化

A better compression driver? CutFEM 3D shape optimization taking viscothermal losses into account

针对压缩驱动器径向相位塞设计易产生共振与干涉的挑战，提出结合粘热边界层模型、水平集与CutFEM的3D形状优化算法，成功使频率响应逼近理想值，有望复兴径向相位塞设计。

2024-03-14cs.NAmath.NAmath.OC

一种用于求解结构化基数惩罚优化问题的带保障加速近端ADMM算法

A Safeguarded Accelerated Proximal ADMM Algorithm for Solving Structured Cardinality Penalized Optimization Problems

School of Mathematics, Harbin Institute of Technology（哈尔滨工业大学数学学院）

本文针对结构化基数惩罚优化问题，提出带保障加速近端ADMM算法，证明其收敛性并建立收敛速率，经信号与图像去噪实验验证了有效性。

2026-10-14math.OC

带欧几里得约束的黎曼quasi-Newton算法

A Riemannian quasi-Newton algorithm for optimization with Euclidean bounds

本文提出一种结合有限记忆quasi-Newton更新和黎曼适应的广义Cauchy点策略的黎曼有限记忆BFGS方法，用于处理欧几里得约束的优化问题，展示了在协方差矩阵估计和神经影像等领域中的高效性。

2026-05-11cs.MSmath.DGmath.OC

Efficient Solution of Discrete Subproblems Arising in Integer Optimal Control with Total Variation Regularization

We consider a class of integer linear programs (IPs) that arise as discretizations of trust-region subproblems of a trust-region algorithm for the solution of control problems, where the control input is an integer-valued function on a one-dimensional domain and is regularized with a total variation term in the objective, which may be interpreted as a penalization of switching costs between different control modes. We prove that solving an instance of the considered problem class is equivalent to solving a resource constrained shortest path problem (RCSPP) on a layered directed acyclic graph. …

2022-06-03math.OC

经验风险最小化中的梯度下降推断

Gradient descent inference in empirical risk minimization

针对梯度下降在高维统计推断中动态理解不足的问题，提出平均场机制下非渐近状态演化理论，刻画迭代与去偏统计量的联合分布，构建去偏梯度下降推断框架，适用于非凸损失和非高斯数据，无需收敛即可实现有效推断。

2025-11-18cs.ITmath.ITmath.OC

超越凸性：用于高效函数约束优化的近端扰动拉格朗日方法

Beyond Convexity: Proximal-Perturbed Lagrangian Methods for Efficient Functional Constrained Optimization

针对非凸函数约束优化问题，提出基于近端扰动增广拉格朗日的原始-对偶算法框架，在温和条件下收敛到稳定点，无需连续调整罚参数，并给出非渐近迭代复杂度及实验验证。

2025-10-28math.OC

Stochastic Optimization for Regularized Wasserstein Estimators

University of Cambridge（剑桥大学）；Google Research（谷歌研究院）；INRIA - DI, ENS, PSL Research University Paris（法国国家信息与自动化研究所 - 数据信息学部、巴黎高等师范学院、巴黎文理研究大学）

Optimal transport is a foundational problem in optimization, that allows to compare probability distributions while taking into account geometric aspects. Its optimal objective value, the Wasserstein distance, provides an important loss between distributions that has been used in many applications throughout machine learning and statistics. Recent algorithmic progress on this problem and its regularized versions have made these tools increasingly popular. However, existing techniques require solving an optimization problem to obtain a single gradient of the loss, thus slowing down first-order …

2020-02-20cs.LGmath.OCstat.ML

Meta-learning based Alternating Minimization Algorithm for Non-convex Optimization

National University of Defense Technology（国防科技大学）；Beihang University（北京航空航天大学）；Imperial College London（帝国理工学院）；University of Modena and Reggio Emilia (UNIMORE)（摩德纳-雷焦艾米利亚大学（UNIMORE））

In this paper, we propose a novel solution for non-convex problems of multiple variables, especially for those typically solved by an alternating minimization (AM) strategy that splits the original optimization problem into a set of sub-problems corresponding to each variable, and then iteratively optimize each sub-problem using a fixed updating rule. However, due to the intrinsic non-convexity of the original optimization problem, the optimization can usually be trapped into spurious local minimum even when each sub-problem can be optimally solved at each iteration. Meanwhile, learning-based …

2022-06-26cs.LGmath.OCstat.ML

从Rashomon集进行差分隐私采样，以及凸优化中Langevin扩散的通用性

Differentially Private Sampling from Rashomon Sets, and the Universality of Langevin Diffusion for Convex Optimization

Google Research（谷歌研究院）；UC Berkeley（加州大学伯克利分校）；Google DeepMind（谷歌DeepMind）；Rutgers University（罗格斯大学）

本研究提出基于Langevin扩散的算法框架，既实现了不依赖凸性、可随时终止的指数机制差分隐私采样及严格一致稳定性保证，得到凸损失下的最优风险界，还可设计Rashomon集的差分隐私一致采样器，支撑可解释、鲁棒等多类机器学习研究。

2023-08-28cs.CRcs.LGmath.OC

具有梯度变化的通用在线学习：一种多层在线集成方法

Universal Online Learning with Gradient Variations: A Multi-layer Online Ensemble Approach

National Key Laboratory for Novel Software Technology（计算机软件新技术国家重点实验室）；Nanjing University（南京大学）

本文提出多层在线集成的通用在线凸优化方法，可在未知函数类型与曲率下利用梯度变化，获得强凸、指数凹和凸情形的问题依赖遗憾界，且每轮仅需一次梯度查询。

2024-04-16cs.LGmath.OCstat.ML