一种简单有效的机器学习张量演算
A Simple and Efficient Tensor Calculus for Machine Learning
摘要：
计算张量表达式的导数，也称为张量演算，是机器学习中的一项基本任务。一个关键的问题是评估表达式及其衍生物的效率，这取决于这些表达式的表示。最近，已经引入了一种用于计算张量表达式（如雅可比矩阵或海森矩阵）的高阶导数的算法，该算法比先前的最先进的方法快几个数量级。不幸的是，该方法基于Ricci符号，因此无法纳入来自深度学习的自动微分框架，如TensorFlow，PyTorch，autograd或JAX，这些框架使用更简单的Einstein符号。这留下了两个选择，要么改变这些框架中的底层张量表示，要么开发一个新的，基于爱因斯坦符号的可证明正确的算法。显然，第一种选择是不切实际的。因此，我们采用第二种选择。在这里，我们表明，使用Ricci符号是没有必要的一个有效的张量演算和开发一个同样有效的方法，更简单的爱因斯坦符号。事实证明，转向爱因斯坦符号可以进一步改进，从而提高效率。本文所描述的方法已在计算矩阵和张量表达式的导数的在线工具www.MatrixCalculus.org中实现。这篇论文的扩展摘要出现在AAAI 2020的“A Simple and Efficient Tensor Calculus”。
英文摘要：
Computing derivatives of tensor expressions, also known as tensor calculus, is a fundamental task in machine learning. A key concern is the efficiency of evaluating the expressions and their derivatives that hinges on the representation of these expressions. Recently, an algorithm for computing higher order derivatives of tensor expressions like Jacobians or Hessians has been introduced that is a few orders of magnitude faster than previous state-of-the-art approaches. Unfortunately, the approach is based on Ricci notation and hence cannot be incorporated into automatic differentiation frameworks from deep learning like TensorFlow, PyTorch, autograd, or JAX that use the simpler Einstein notation. This leaves two options, to either change the underlying tensor representation in these frameworks or to develop a new, provably correct algorithm based on Einstein notation. Obviously, the first option is impractical. Hence, we pursue the second option. Here, we show that using Ricci notation is not necessary for an efficient tensor calculus and develop an equally efficient method for the simpler Einstein notation. It turns out that turning to Einstein notation enables further improvements that lead to even better efficiency. The methods that are described in this paper have been implemented in the online tool www.MatrixCalculus.org for computing derivatives of matrix and tensor expressions. An extended abstract of this paper appeared as "A Simple and Efficient Tensor Calculus", AAAI 2020.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
