arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2006.06903cs.LGstat.ML
收藏

不可微函数自动微分的正确性

On Correctness of Automatic Differentiation for Non-Differentiable Functions

KAIST(韩国科学技术院);INRIA Paris(法国国家信息与自动化研究所巴黎分部);ENS(巴黎高等师范学院);CNRS/PSL University(法国国家科学研究中心/巴黎文理研究大学)

Wonyeol Lee, Hangyeol Yu, Xavier Rival, Hongseok Yang

摘要:

差异化是许多机器学习算法的核心,并得到流行的autodiff系统(如TensorFlow和PyTorch)的良好支持。最初,这些系统是为了计算可微函数的导数而开发的,但在实践中,它们通常应用于不可微函数。例如,使用ReLU的神经网络通常定义不可微函数,但实际上使用autodiff系统计算涉及这些函数的损失梯度。这种现状提出了一个自然的问题:autodiff系统在应用于这种不可微函数时,在任何形式意义上都是正确的吗?在本文中,我们提供了一个肯定的答案,这个问题。使用反例,我们首先指出经常使用的非正式论点中的缺陷,例如:深度学习中出现的不可微性不会引起任何问题,因为它们形成了一个测量零集。然后,我们研究了一类称为PAP函数的函数,它几乎包括了当今深度学习中的所有(可能是不可微的)函数。对于这些PAP函数,我们提出了一种新的衍生物,称为内涵衍生物,并证明了这些衍生物总是存在的,并符合几乎所有的输入标准的衍生物。我们还表明,这些内涵的衍生物是大多数autodiff系统计算或试图计算基本上。通过这种方式,我们正式建立了autodiff系统应用于不可微函数的正确性。

英文摘要:

Differentiation lies at the core of many machine-learning algorithms, and is well-supported by popular autodiff systems, such as TensorFlow and PyTorch. Originally, these systems have been developed to compute derivatives of differentiable functions, but in practice, they are commonly applied to functions with non-differentiabilities. For instance, neural networks using ReLU define non-differentiable functions in general, but the gradients of losses involving those functions are computed using autodiff systems in practice. This status quo raises a natural question: are autodiff systems correct in any formal sense when they are applied to such non-differentiable functions? In this paper, we provide a positive answer to this question. Using counterexamples, we first point out flaws in often-used informal arguments, such as: non-differentiabilities arising in deep learning do not cause any issues because they form a measure-zero set. We then investigate a class of functions, called PAP functions, that includes nearly all (possibly non-differentiable) functions in deep learning nowadays. For these PAP functions, we propose a new type of derivatives, called intensional derivatives, and prove that these derivatives always exist and coincide with standard derivatives for almost all inputs. We also show that these intensional derivatives are what most autodiff systems compute or try to compute essentially. In this way, we formally establish the correctness of autodiff systems applied to non-differentiable functions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑