arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1711.01348cs.SCstat.ML
收藏

张量代数的自动微分

Automatic Differentiation for Tensor Algebras

Sebastian Urban, Patrick van der Smagt

摘要:

Kjolstad et. al.提出了一个张量代数编译器。它采用按元素定义张量的表达式(如$f_{ij}(a,b,c,d) = \exp\left[-\sum_{k=0}^4 \left((a_{ik}+b_{jk})^2\, c_{ii} + d_{i+k}^3 \right) \right]$),并生成相应的计算内核代码。对于机器学习,特别是深度学习,通常需要计算损失函数$l(a,b,c,d)=l(f(a,b,c,d))$相对于参数$a,b,c,d$的梯度。如果张量编译器要应用在这个领域,它是必要的推导表达式的衍生物的元素定义的张量,即表达式的$(da)_{ik}=\partial l/\partial a_{ik}$。当函数索引和参数索引之间的映射不是1:1时,需要特别注意。对于函数$f_{ij} (x) = x_i^2$,损失的导数是$(dx)_i=\partial l/\partial x_i=\sum_j (df)_{ij}2x_i$ ;求和是必要的,因为索引$j$不出现在$f$的索引中。另一个例子是$f_{i}(x)=x_{ii}^2$,其中$x$是一个矩阵;这里我们有$(dx)_{ij}=δ_{ij}(df)_i2x_{ii}$ ;克罗内克增量是必要的,因为非对角元素的导数为零。$f_{ij}(x)=\exp x_{i+j}$使用另一种索引方案;这里正确的导数是$(dx)_{k}=\sum_i (df)_{i,k-i} \exp x_{k}$,其中必须适当选择和的范围。在本出版物中,我们提出了一种算法,可以处理任何情况下,其中参数的索引是函数的索引的任意线性组合,因此可以处理所有上述示例。和(及其范围)和克罗内克增量会根据需要自动插入到导数中。此外,如果需要,索引会被转换(如上一个示例)。该算法输出一个符号表达式,可以随后送入张量代数编译器。提供了源代码。

英文摘要:

Kjolstad et. al. proposed a tensor algebra compiler. It takes expressions that define a tensor element-wise, such as $f_{ij}(a,b,c,d) = \exp\left[-\sum_{k=0}^4 \left((a_{ik}+b_{jk})^2\, c_{ii} + d_{i+k}^3 \right) \right]$, and generates the corresponding compute kernel code. For machine learning, especially deep learning, it is often necessary to compute the gradient of a loss function $l(a,b,c,d)=l(f(a,b,c,d))$ with respect to parameters $a,b,c,d$. If tensor compilers are to be applied in this field, it is necessary to derive expressions for the derivatives of element-wise defined tensors, i.e. expressions for $(da)_{ik}=\partial l/\partial a_{ik}$. When the mapping between function indices and argument indices is not 1:1, special attention is required. For the function $f_{ij} (x) = x_i^2$, the derivative of the loss is $(dx)_i=\partial l/\partial x_i=\sum_j (df)_{ij}2x_i$; the sum is necessary because index $j$ does not appear in the indices of $f$. Another example is $f_{i}(x)=x_{ii}^2$, where $x$ is a matrix; here we have $(dx)_{ij}=δ_{ij}(df)_i2x_{ii}$; the Kronecker delta is necessary because the derivative is zero for off-diagonal elements. Another indexing scheme is used by $f_{ij}(x)=\exp x_{i+j}$; here the correct derivative is $(dx)_{k}=\sum_i (df)_{i,k-i} \exp x_{k}$, where the range of the sum must be chosen appropriately. In this publication we present an algorithm that can handle any case in which the indices of an argument are an arbitrary linear combination of the indices of the function, thus all the above examples can be handled. Sums (and their ranges) and Kronecker deltas are automatically inserted into the derivatives as necessary. Additionally, the indices are transformed, if required (as in the last example). The algorithm outputs a symbolic expression that can be subsequently fed into a tensor algebra compiler. Source code is provided.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑