区分张量语言
Differentiating a Tensor Language
摘要:
如何编译张量程序的衍生程序,使得生成的代码是纯功能性的(因此更容易优化和并行化),并且相对于原始程序可证明是有效的?我们发现,天真地区分张量代码--就像在Tensorflow和PyTorch等流行系统中所做的那样--可能会在病理情况下导致渐近减速,违反廉价的先决条件原则。然而,所有现有的自动微分方法,保证这一原则(可变大小的数据)这样做,依靠+=突变通过别名/指针-这复杂的下游优化。我们提供了第一个纯功能的,可证明有效的,伴随/反向模式衍生物的数组/张量代码明确占稀疏性。我们通过关注艾弗森的APL的指标函数来做到这一点。我们还介绍了一个新的“张量SSA”范式和一个新的推导反向模式自动微分的内积的基础上的通用性质。
英文摘要:
How does one compile derivatives of tensor programs, such that the resulting code is purely functional (hence easier to optimize and parallelize) and provably efficient relative to the original program? We show that naively differentiating tensor code---as done in popular systems like Tensorflow and PyTorch---can cause asymptotic slowdowns in pathological cases, violating the Cheap Gradients Principle. However, all existing automatic differentiation methods that guarantee this principle (for variable size data) do so by relying on += mutation through aliases/pointers---which complicates downstream optimization. We provide the first purely functional, provably efficient, adjoint/reverse-mode derivatives of array/tensor code by explicitly accounting for sparsity. We do this by focusing on the indicator function from Iverson's APL. We also introduce a new "Tensor SSA" normal form and a new derivation of reverse-mode automatic differentiation based on the universal property of inner-products.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。