在 Ricci 流下学习离散化神经网络
Learning Discretized Neural Networks under Ricci Flow
AI总结：
针对离散化神经网络中STE导致的梯度不匹配问题，本文基于信息几何构建LNE流形并引入Ricci流，证明度量扰动呈指数衰减，显著提升了网络训练的稳定性和性能。
摘要：
本文研究了由低精度权重和激活组成的离散化神经网络（DNNs），由于训练期间不可微的离散函数，它们会遭受无限或零梯度的问题。此类场景中大多数基于训练的 DNNs 采用标准 Straight-Through Estimator (STE) 来近似关于离散值的梯度。然而，使用 STE 引入了由近似梯度扰动引起的梯度不匹配问题。为解决此问题，本文揭示这种不匹配可以通过对偶理论的视角解释为黎曼流形上的度量扰动。基于信息几何，我们为 DNNs 构建了 Linearly Nearly Euclidean (LNE) 流形，为解决扰动提供了背景。通过在度量上引入偏微分方程，即 Ricci 流，我们建立了具有 $L^2$-范数扰动的 LNE 度量的动态稳定性和收敛性。与以往收敛率为分数幂的扰动理论相比，Ricci 流下的度量扰动在 LNE 流形中表现出指数衰减。跨多个数据集的实验结果表明，与其他代表性的基于训练的方法相比，我们的方法为 DNNs 实现了更优且更稳定的性能。
英文摘要：
In this paper, we study Discretized Neural Networks (DNNs) composed of low-precision weights and activations, which suffer from either infinite or zero gradients due to the non-differentiable discrete function during training. Most training-based DNNs in such scenarios employ the standard Straight-Through Estimator (STE) to approximate the gradient w.r.t. discrete values. However, the use of STE introduces the problem of gradient mismatch, arising from perturbations in the approximated gradient. To address this problem, this paper reveals that this mismatch can be interpreted as a metric perturbation in a Riemannian manifold, viewed through the lens of duality theory. Building on information geometry, we construct the Linearly Nearly Euclidean (LNE) manifold for DNNs, providing a background for addressing perturbations. By introducing a partial differential equation on metrics, i.e., the Ricci flow, we establish the dynamical stability and convergence of the LNE metric with the $L^2$-norm perturbation. In contrast to previous perturbation theories with convergence rates in fractional powers, the metric perturbation under the Ricci flow exhibits exponential decay in the LNE manifold. Experimental results across various datasets demonstrate that our method achieves superior and more stable performance for DNNs compared to other representative training-based methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
