arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.13370cs.LGstat.ML
收藏

关于具有机器可表示参数的神经网络自动微分正确性的研究

On the Correctness of Automatic Differentiation for Neural Networks with Machine-Representable Parameters

Stanford University(斯坦福大学);Korea University(高丽大学)

Wonyeol Lee, Sejun Park, Alex Aiken

AI总结:

本文针对机器可表示参数下的神经网络自动微分,证明带偏置网络的 AD 不正确集为空,刻画不可分集的线性紧界,并说明 AD 在不可微点仍给出 Clarke 次导数。

摘要:

近期工作表明,实数上的前向模式和反向模式自动微分(AD)在数学精确意义下几乎总是正确的。然而,实际程序处理的是机器可表示数(例如浮点数),而非实数。本文研究当神经网络的参数空间完全由机器可表示数组成时 AD 的正确性。具体而言,我们分析了两组可能导致 AD 不正确的参数:不正确集,即网络在该集合上可微但 AD 未计算其导数;以及不可分集,即网络在该集合上不可微。对于带偏置参数的神经网络,我们首先证明不正确集始终为空。随后,我们证明了不可分集大小的紧界,其与激活函数中不可微点的数量呈线性关系,并给出了参数属于该集合的简单充要条件。我们进一步证明,即使在不可分集上,AD 也始终计算 Clarke 次导数。我们还将这些结果扩展到可能没有偏置参数的神经网络。

英文摘要:

Recent work has shown that forward- and reverse- mode automatic differentiation (AD) over the reals is almost always correct in a mathematically precise sense. However, actual programs work with machine-representable numbers (e.g., floating-point numbers), not reals. In this paper, we study the correctness of AD when the parameter space of a neural network consists solely of machine-representable numbers. In particular, we analyze two sets of parameters on which AD can be incorrect: the incorrect set on which the network is differentiable but AD does not compute its derivative, and the non-differentiable set on which the network is non-differentiable. For a neural network with bias parameters, we first prove that the incorrect set is always empty. We then prove a tight bound on the size of the non-differentiable set, which is linear in the number of non-differentiabilities in activation functions, and give a simple necessary and sufficient condition for a parameter to be in this set. We further prove that AD always computes a Clarke subderivative even on the non-differentiable set. We also extend these results to neural networks possibly without bias parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑