随机学习动力学中信息获取的速度极限
Speed Limit for Information Acquisition in Stochastic Learning Dynamics
浏览论文内容
中文总结 AI 辅助
本研究将SGD建模为马尔可夫过程，推导Fisher信息流速度极限，量化信息获取速率，并在线性回归中验证其预测能力。
中文摘要 AI 辅助
神经网络通过学习获取内部表征。在本工作中，我们将随机梯度下降（SGD）表述为一个马尔可夫随机过程，并推导出一个Fisher信息流速度极限，该极限限制了可训练参数从数据生成过程中的潜在变量获取信息的速率。所得不等式将信息流分解为漂移项和噪声项，从而从信息论角度量化了确定性学习力和SGD诱导波动的作用。我们在解析可处理的基函数线性回归中验证了该界限，其中该界限预测的信息预算复现了不同潜在变量在学习参数中被编码的顺序和特征时间尺度。这些结果确立了Fisher信息速度极限作为一个定量框架，用于诊断随机学习过程中数据生成机制的不同方面何时以及如何被获取。
英文摘要
Neural networks acquire internal representations through learning. In this work, we formulate stochastic gradient descent (SGD) as a Markovian stochastic process and derive a Fisher-information flow speed limit that bounds the rate at which trainable parameters can acquire information about latent variables in the data-generating process. The resulting inequality decomposes the information flow into drift and noise contributions, thereby quantifying the roles of deterministic learning forces and SGD-induced fluctuations from an information-theoretic perspective. We verify the bound in analytically tractable basis-function linear regression, where the information budget predicted by the bound reproduces the ordering and characteristic time scales with which different latent variables are encoded in the learned parameters. These results establish Fisher-information speed limits as a quantitative framework for diagnosing when and how different aspects of the data-generating mechanism are acquired during stochastic learning.
发表机构
- Kyoto University（京都大学）
机构由 AI 辅助整理，请以论文原文为准。
