退火在InfoNCE中的渐近性和有限时间保证

CoolMomentum: A Method for Stochastic Optimization by Langevin Dynamics with Simulated Annealing

Deep learning applications require global optimization of non-convex objective functions, which have multiple local minima. The same problem is often found in physical simulations and may be resolved by the methods of Langevin dynamics with Simulated Annealing, which is a well-established approach for minimization of many-particle potentials. This analogy provides useful insights for non-convex stochastic optimization in machine learning. Here we find that integration of the discretized Langevin equation gives a coordinate updating rule equivalent to the famous Momentum optimization algorithm.…

Convex Optimization with Unbounded Nonconvex Oracles using Simulated Annealing

We consider the problem of minimizing a convex objective function $F$ when one can only evaluate its noisy approximation $\hat{F}$. Unless one assumes some structure on the noise, $\hat{F}$ may be an arbitrary nonconvex function, making the task of minimizing $F$ intractable. To overcome this, prior work has often focused on the case when $F(x)-\hat{F}(x)$ is uniformly-bounded. In this paper we study the more general case when the noise has magnitude $αF(x) + β$ for some $α, β> 0$, and present a polynomial time algorithm that finds an approximate minimizer of $F$ for this noise model. Previous…

Uniform Generalization Bound on Time and Inverse Temperature for Gradient Descent Algorithm and its Application to Analysis of Simulated Annealing

In this paper, we propose a novel uniform generalization bound on the time and inverse temperature for stochastic gradient Langevin dynamics (SGLD) in a non-convex setting. While previous works derive their generalization bounds by uniform stability, we use Rademacher complexity to make our generalization bound independent of the time and inverse temperature. Using Rademacher complexity, we can reduce the problem to derive a generalization bound on the whole space to that on a bounded region and therefore can remove the effect of the time and inverse temperature from our generalization bound. …

正则化 Langevin 动力学用于组合优化

Regularized Langevin Dynamics for Combinatorial Optimization

本文提出正则化 Langevin 动力学用于组合优化，通过避免局部极小值提升传统启发式方法和神经网络模型的性能。

时间非齐次随机微分方程的Fisher信息耗散

Fisher information dissipation for time inhomogeneous stochastic differential equations

该研究针对时间非齐次变系数随机微分方程开展Lyapunov收敛分析，通过修正梯度流表述、选取时间依赖相对Fisher信息泛函并提出Hessian矩阵条件，证明了三类典型Langevin动力学的收敛性并通过数值算例验证。

关于组成式得分方法在基于模拟的推断中的退火动力学理论指南

Theoretical guidelines for annealed Langevin dynamics in compositional simulation-based inference

本文研究了基于模拟的推断中组成式得分方法的退火动力学理论，提出了一种新的理论框架，通过推导Wasserstein界，为超参数选择提供了理论指导，并在高斯情况下证明了不同复合得分方法在步长和总动力学步数上的差异。