基于自然梯度的变分自回归网络高效优化
Efficient Optimization of Variational Autoregressive Networks with Natural Gradient
AI总结:
针对变分自回归网络(VAN)优化变分自由能时的计算挑战,提出基于自然梯度的ng-VAN框架,其计算复杂度与批大小三次方相关,在多个统计力学模型上实验显示,该方法相比传统VAN显著提升自由能估计准确性、加快收敛速度,扩展了VAN的应用范围。
摘要:
估计自由能是统计力学中的一个基本问题。近年来,基于机器学习的方法,特别是变分自回归网络(VANs)被提出,用于最小化变分自由能并近似玻尔兹曼分布。VAN具有显著优势,包括可精确计算归一化联合分布和快速采样,这些是马尔可夫链蒙特卡罗算法中常缺失的关键特征。然而,VAN也面临重大计算挑战,包括在复杂参数空间中优化变分自由能的困难以及学习收敛缓慢。在这项工作中,我们将基于自然梯度的优化技术引入VAN框架,即ng-VAN,以提高传统VAN的学习效率和准确性。该方法的计算复杂度与批大小的三次方成正比,而非模型参数数量的三次方,因此可高效应用于大型VAN模型。我们在Sherrington-Kirkpatrick模型、随机图上的自旋玻璃以及二维伊辛模型上进行了广泛的数值实验。结果表明,与传统VAN相比,ng-VAN显著提高了自由能估计的准确性,收敛速度更快且学习时间更短。这使得VAN框架能够应用于此前无法处理的具有挑战性的统计力学问题。
英文摘要:
Estimating free energy is a fundamental problem in statistical mechanics. Recently, machine-learning-based methods, particularly the variational autoregressive networks (VANs) have been proposed to minimize variational free energy and to approximate the Boltzmann distribution. VAN enjoys notable advantages, including the exact computation of the normalized joint distribution and fast sampling, which are critical features often missing in Markov chain Monte Carlo algorithms. However, VAN also faces significant computational challenges. These include difficulties in the optimization of variational free energy in a complicated parameter space and slow convergence of learning. In this work, we introduce an optimization technique based on natural gradients to the VAN framework, namely ng-VAN, to enhance the learning efficiency and accuracy of the conventional VAN. The method has computational complexity cubic in the batch size rather than in the number of model parameters, hence it can be efficiently implemented for a large VAN model. We carried out extensive numerical experiments on the Sherrington-Kirkpatrick model, spin glasses on random graphs, and the two-dimensional Ising model. Our results indicate that compared with the conventional VAN, ng-VAN significantly improves the accuracy in estimating free energy and converges much faster with shorter learning time. This allows extending the VAN framework's applicability to challenging statistical mechanics problems that were previously not accessible.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。