Softmax不足以实现鲁棒的规模泛化
Softmax is not Enough (for Sharp Size Generalisation)
AI总结:
本文揭示了softmax函数在问题规模增加时无法鲁棒近似明确函数的理论局限,证明其电路会随输入增多而分散,并提出自适应温度技术以改善推理时的明确性。
摘要:
推理系统的一个关键特性是能够对其输入数据做出明确的决策。对于当代AI系统而言,明确行为的关键载体是softmax函数,它具备执行可微查询-键查找的能力。人们普遍认为,利用softmax的网络的预测能力源于“电路”,这些电路能够在许多不同的输入上一致地明确执行某些类型的计算。然而,要使这些电路具有鲁棒性,它们需要能够很好地泛化到任意有效的输入。在本文中,我们打破了这一迷思:即使对于寻找最大键这样简单的任务,任何学习到的电路在测试时随着项目数量的增加也必然会发生分散。我们将此归因于softmax函数在问题规模增加时鲁棒地近似明确函数的根本局限性,在理论上证明了这一现象,并提出了自适应温度作为一种临时技术,用于在推理时改善softmax的明确性。
英文摘要:
A key property of reasoning systems is the ability to make sharp decisions on their input data. For contemporary AI systems, a key carrier of sharp behaviour is the softmax function, with its capability to perform differentiable query-key lookups. It is a common belief that the predictive power of networks leveraging softmax arises from "circuits" which sharply perform certain kinds of computations consistently across many diverse inputs. However, for these circuits to be robust, they would need to generalise well to arbitrary valid inputs. In this paper, we dispel this myth: even for tasks as simple as finding the maximum key, any learned circuitry must disperse as the number of items grows at test time. We attribute this to a fundamental limitation of the softmax function to robustly approximate sharp functions with increasing problem size, prove this phenomenon theoretically, and propose adaptive temperature as an ad-hoc technique for improving the sharpness of softmax at inference time.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。