两遍Softmax算法
The Two-Pass Softmax Algorithm
摘要：
softmax（也称为softargmax）函数广泛用于机器学习模型中，将实值分数归一化为概率分布。为了避免浮点溢出，softmax函数通常在三个过程中实现：第一个过程用于计算归一化常数，另外两个过程用于从归一化输入计算输出。我们分析了两个变种的三遍算法，并证明在一个良好的优化实现HPC级处理器的性能，所有三个通行证是有限的内存带宽。然后，我们提出了一种新的算法softmax计算在短短两个通道。所提出的两遍算法通过采用中间值的奇异表示来避免数值溢出和额外的归一化过程，其中每个值被表示为一对浮点数：一个表示“尾数”，另一个表示“指数”。性能评估表明，在英特尔Skylake-X处理器的缓存外输入上，新的两遍算法在AVX 512实施中比传统的三遍算法性能高出28%，在AVX 2实施中比传统的三遍算法性能高出18%。在Intel Broadwell和AMD Zen 2处理器上，提出的Two-Pass算法也优于传统的Three-Pass算法。为了促进可重复性，我们在本文中发布了新的Two-Pass Softmax算法和其他实验的开源实现，作为XNNPACK库的一部分，网址为GitHub.com/google/XNNPACK。
英文摘要：
The softmax (also called softargmax) function is widely used in machine learning models to normalize real-valued scores into a probability distribution. To avoid floating-point overflow, the softmax function is conventionally implemented in three passes: the first pass to compute the normalization constant, and two other passes to compute outputs from normalized inputs. We analyze two variants of the Three-Pass algorithm and demonstrate that in a well-optimized implementation on HPC-class processors performance of all three passes is limited by memory bandwidth. We then present a novel algorithm for softmax computation in just two passes. The proposed Two-Pass algorithm avoids both numerical overflow and the extra normalization pass by employing an exotic representation for intermediate values, where each value is represented as a pair of floating-point numbers: one representing the "mantissa" and another representing the "exponent". Performance evaluation demonstrates that on out-of-cache inputs on an Intel Skylake-X processor the new Two-Pass algorithm outperforms the traditional Three-Pass algorithm by up to 28% in AVX512 implementation, and by up to 18% in AVX2 implementation. The proposed Two-Pass algorithm also outperforms the traditional Three-Pass algorithm on Intel Broadwell and AMD Zen 2 processors. To foster reproducibility, we released an open-source implementation of the new Two-Pass Softmax algorithm and other experiments in this paper as a part of XNNPACK library at GitHub.com/google/XNNPACK.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
