发表机构
KIIT（卡林加工业技术学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究Transformer中归一化策略选择问题，通过实验发现不同任务中静态与自适应归一化的表现差异，提出AutoNorm-S训练策略，通过门控冻结计划减轻优化不稳定性，在多基准测试中取得良好性能。
AI 中文摘要
归一化是稳定Transformer训练的关键组件，但诸如层归一化（LN）等静态策略与自适应方法之间的选择很大程度上仍取决于任务。本文研究了可微归一化门控中的一个关键优化挑战。实验表明，在相对稳定的视觉任务上，Gumbel-Softmax门控引入的高梯度方差会阻碍路由机制收敛，使学习到的门控表现不如简单随机选择。而在非稳定的语言建模和分类任务上，持续的门控多样性能让模型学习到更有效的逐层归一化策略。基于此，提出AutoNorm-S（稳定版）训练策略，通过门控冻结计划减轻优化不稳定性。AutoNorm-S在多个基准测试中取得了有竞争力或更好的性能，在NLP数据集上优于自适应归一化基线，在标准视觉基准测试中也具有竞争力。这些结果表明，将归一化选择与优化噪声解耦为Transformer架构中的自适应归一化提供了一种实用且有原则的方法。
英文摘要
Normalization is a critical component for stabilizing Transformer training, yet the choice between static strategies such as Layer Normalization (LN) and adaptive alternatives remains largely task-dependent. In this paper, we investigate a key optimization challenge in differentiable normalization gating. Our experiments show that, on relatively stationary vision tasks, the high gradient variance introduced by Gumbel-Softmax gating can hinder convergence of the routing mechanism, causing learned gates to underperform simple random selection. In contrast, on non-stationary language modeling and classification tasks, sustained gating diversity enables the model to learn more effective layer-wise normalization policies. Motivated by these observations, we propose AutoNorm-S (Stabilized), a training strategy that mitigates optimization instability through a gate-freezing schedule. AutoNorm-S achieves competitive or improved performance across multiple benchmarks, outperforming adaptive normalization baselines on NLP datasets, including PTB and SST-2, while remaining competitive on standard vision benchmarks. These results suggest that decoupling normalization selection from optimization noise provides a practical and principled approach for adaptive normalization in Transformer architectures.
