利用交换Langevin扩散加速非凸学习
Accelerating Nonconvex Learning via Replica Exchange Langevin Diffusion
摘要：
朗之万扩散是非凸优化的一种强大方法，它可以通过向梯度中注入噪声来摆脱局部最小值。特别地，控制噪声水平的温度参数引起"全局探测“和"局部利用”之间的权衡，这对应于高温和低温。为了实现这两个制度的优点，我们建议使用副本交换，交换两个朗之万扩散与不同的温度。本文从两个方面对副本交换的加速效应进行了理论分析：（i）χ^2-发散下的收敛性;（ii）大偏差原理。这种加速效应使我们能够更快地接近全局最小值。此外，通过离散化副本交换Langevin扩散，我们得到了一个离散时间算法。对于这种算法，我们从理论上量化了它的离散化误差，并在实践中证明了它的加速效果。
英文摘要：
Langevin diffusion is a powerful method for nonconvex optimization, which enables the escape from local minima by injecting noise into the gradient. In particular, the temperature parameter controlling the noise level gives rise to a tradeoff between ``global exploration'' and ``local exploitation'', which correspond to high and low temperatures. To attain the advantages of both regimes, we propose to use replica exchange, which swaps between two Langevin diffusions with different temperatures. We theoretically analyze the acceleration effect of replica exchange from two perspectives: (i) the convergence in χ^2-divergence, and (ii) the large deviation principle. Such an acceleration effect allows us to faster approach the global minima. Furthermore, by discretizing the replica exchange Langevin diffusion, we obtain a discrete-time algorithm. For such an algorithm, we quantify its discretization error in theory and demonstrate its acceleration effect in practice.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
