arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.06008cs.LGcond-mat.dis-nnhep-thstat.ML
收藏

神经网络高斯过程的 Wilsonian 重整化

Wilsonian Renormalization of Neural Network Gaussian Processes

Kavli Institute for Theoretical Physics, Santa Barbara, CA USA(卡弗里理论物理研究所);Institute for Theoretical Physics, and Department of Information and Computing Sciences Utrecht University(理论物理研究所及信息与计算科学系,乌特雷赫大学);Perimeter Institute for Theoretical Physics(皮尔敏研究所);The Racah Institute of Physics, The Hebrew University of Jerusalem(拉赫理论物理研究所,耶路撒冷希伯来大学)

Jessica N. Howard, Ro Jefferson, Anindita Maiti, Zohar Ringel

AI总结:

本文提出在 GP 回归中执行 Wilsonian RG,通过积分核的不可学习模式得到由数据设定 IR 尺度的 RG 流,以连接 RG 与神经网络特征学习。

摘要:

区分相关信息与无关信息是任何建模过程或科学探究的关键。理论物理学以重整化群(RG)的形式为实现这一点提供了强大工具。在此,我们展示了一种在 Gaussian Process(GP,高斯过程)回归背景下执行 Wilsonian RG 的实用方法。我们系统地积分掉 GP 核中不可学习的模式,从而获得一个由数据集设定 IR 尺度的 GP 的 RG 流。在简单情形下,这会产生 ridge 参数的普适流;而在包含非高斯性的更丰富情形中,该流会变得依赖于输入。除了在解析上易于处理之外,该方法还超越了 RG 与神经网络之间的结构类比,在 RG 流与可学习及不可学习模式之间建立了自然联系。研究此类流可能增进我们对深度神经网络中特征学习的理解,并使我们能够识别这些模型中潜在的普适性类别。

英文摘要:

Separating relevant and irrelevant information is key to any modeling process or scientific inquiry. Theoretical physics offers a powerful tool for achieving this in the form of the renormalization group (RG). Here we demonstrate a practical approach to performing Wilsonian RG in the context of Gaussian Process (GP) Regression. We systematically integrate out the unlearnable modes of the GP kernel, thereby obtaining an RG flow of the GP in which the data sets the IR scale. In simple cases, this results in a universal flow of the ridge parameter, which becomes input-dependent in the richer scenario in which non-Gaussianities are included. In addition to being analytically tractable, this approach goes beyond structural analogies between RG and neural networks by providing a natural connection between RG flow and learnable vs. unlearnable modes. Studying such flows may improve our understanding of feature learning in deep neural networks, and enable us to identify potential universality classes in these models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑