arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2204.12939cond-mat.dis-nncs.LGhep-thstat.ML
收藏

论推理与学习的动力学

On the Dynamics of Inference and Learning

Queen Mary University of London(伦敦玛丽女王大学);University of Pennsylvania(宾夕法尼亚大学);University of Illinois(伊利诺伊大学)

David S. Berman, Jonathan J. Heckman, Marc Klinger

摘要:

统计推断是确定给定数据集的模型参数空间上的概率分布的过程。随着更多的数据变得可用,通过应用贝叶斯定理,该概率分布变得更新。我们提出了一个治疗的贝叶斯更新过程作为一个连续的动态系统。然后,统计推断由一阶微分方程控制,该微分方程描述由参数模型族确定的信息几何中的轨迹或流。我们解决这个方程的一些简单的模型,并表明,当Cramér-Rao界饱和的学习率是由一个简单的$1/T$幂律,与$T$的时间变量表示的数据量。隐变量的存在可以被合并到该设置中,从而导致在所得到的流方程中的附加驱动项。我们说明了这与分析和数值例子的基础上高斯和高斯随机过程和推断的耦合常数在一维伊辛模型。最后,我们比较贝叶斯流所表现出的定性行为,以各种神经网络的基准数据集,如MNIST和CIFAR 10的训练,并显示如何表现出小的最终损失的网络简单的幂律也得到满足。

英文摘要:

Statistical Inference is the process of determining a probability distribution over the space of parameters of a model given a data set. As more data becomes available this probability distribution becomes updated via the application of Bayes' theorem. We present a treatment of this Bayesian updating process as a continuous dynamical system. Statistical inference is then governed by a first order differential equation describing a trajectory or flow in the information geometry determined by a parametric family of models. We solve this equation for some simple models and show that when the Cramér-Rao bound is saturated the learning rate is governed by a simple $1/T$ power-law, with $T$ a time-like variable denoting the quantity of data. The presence of hidden variables can be incorporated in this setting, leading to an additional driving term in the resulting flow equation. We illustrate this with both analytic and numerical examples based on Gaussians and Gaussian Random Processes and inference of the coupling constant in the 1D Ising model. Finally we compare the qualitative behaviour exhibited by Bayesian flows to the training of various neural networks on benchmarked data sets such as MNIST and CIFAR10 and show how that for networks exhibiting small final losses the simple power-law is also satisfied.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑