更强的无归一化变换器
Stronger Normalization-Free Transformers
AI总结：
本文提出Derf函数，通过大规模搜索找到更有效的归一化替代方案，其在视觉识别、语音表示和DNA序列建模中均优于LayerNorm、RMSNorm和DyT。
摘要：
尽管归一化层长期以来被视为深度学习架构不可或缺的组成部分，但动态双曲正切（DyT）的引入表明替代方案是可能的。点式函数DyT通过约束极端值以实现稳定收敛，并达到归一化级别的性能；本文进一步寻找能超越它的函数设计。我们首先研究了点式函数的内在属性如何影响训练和性能。基于这些发现，我们进行大规模搜索以寻找更有效的函数设计。通过探索，我们引入Derf(x)=erf(αx+s)，其中erf(x)是归一化高斯累积分布函数，并将其识别为最有效的设计。Derf在视觉识别和生成、语音表示和DNA序列建模等多个领域均优于LayerNorm、RMSNorm和DyT。我们的分析还表明，Derf的性能提升主要归因于其改进的泛化能力，而非更强的拟合能力。其简单性和更强的性能使其成为无归一化变换器架构的实用选择。
英文摘要：
Although normalization layers have long been viewed as indispensable components of deep learning architectures, the recent introduction of Dynamic Tanh (DyT) has demonstrated that alternatives are possible. The point-wise function DyT constrains extreme values for stable convergence and reaches normalization-level performance; this work seeks further for function designs that can surpass it. We first study how the intrinsic properties of point-wise functions influence training and performance. Building on these findings, we conduct a large-scale search for a more effective function design. Through this exploration, we introduce $\mathrm{Derf}(x) = \mathrm{erf}(αx + s)$, where $\mathrm{erf}(x)$ is the rescaled Gaussian cumulative distribution function, and identify it as the most performant design. Derf outperforms LayerNorm, RMSNorm, and DyT across a wide range of domains, including visual recognition and generation, speech representation, and DNA sequence modeling. Our analysis also suggests that the performance gains of Derf largely stem from its improved generalization rather than stronger fitting capacity. Its simplicity and stronger performance make Derf a practical choice for normalization-free Transformer architectures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
