Log-Sum-Exp函数和Softmax函数的精确计算
Accurate Computation of the Log-Sum-Exp and Softmax Functions
摘要：
计算log-sum-exp函数或softmax函数是许多现代数据科学算法中的关键步骤，特别是在推理和分类中。由于这些函数包含指数，因此计算容易上溢和下溢，特别是在低精度算术中。软件实现通常使用替代公式，以避免上溢并减少有害下溢的机会，采用移位或另一种重写。虽然在数学上是等价的，但这些变体在浮点运算中的行为不同。我们给舍入误差分析不同的评估算法和解释的误差界使用条件数的功能。我们的结论是，基于分析和数值实验，移位的公式是类似的准确性，未移位的，移位的softmax公式通常比一个无除法的变体更准确。
英文摘要：
Evaluating the log-sum-exp function or the softmax function is a key step in many modern data science algorithms, notably in inference and classification. Because of the exponentials that these functions contain, the evaluation is prone to overflow and underflow, especially in low precision arithmetic. Software implementations commonly use alternative formulas that avoid overflow and reduce the chance of harmful underflow, employing a shift or another rewriting. Although mathematically equivalent, these variants behave differently in floating-point arithmetic. We give rounding error analyses of different evaluation algorithms and interpret the error bounds using condition numbers for the functions. We conclude, based on the analysis and numerical experiments, that the shifted formulas are of similar accuracy to the unshifted ones and that the shifted softmax formula is typically more accurate than a division-free variant.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
