GeoNorm: 通过测地优化统一预归一化与后归一化
GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization
AI总结：
GeoNorm通过测地优化统一预归一化与后归一化，提升Transformer模型性能。
摘要：
在Transformer架构设计中，归一化层的位置，特别是预归一化和后归一化，仍是一个开放性问题。在本工作中，我们通过流形优化的视角重新思考这些方法，将前馈网络（FFN）和注意力层的输出解释为优化中的更新方向。基于这一视角，我们引入GeoNorm，一种新的方法，用流形上的测地更新替代标准归一化。此外，类似于学习率调度，我们为FFN和注意力组件提出了一种层间更新衰减。全面的实验表明，GeoNorm在Transformer模型中始终优于现有的归一化方法。关键的是，GeoNorm可以无缝集成到标准的Transformer架构中，以极小的额外计算成本实现性能提升。
英文摘要：
The placement of normalization layers, specifically Pre-Norm and Post-Norm, remains an open question in Transformer architecture design. In this work, we rethink these approaches through the lens of manifold optimization, interpreting the outputs of the Feed-Forward Network (FFN) and attention layers as update directions in optimization. Building on this perspective, we introduce GeoNorm, a novel method that replaces standard normalization with geodesic updates on the manifold. Furthermore, analogous to learning rate schedules, we propose a layer-wise update decay for the FFN and attention components. Comprehensive experiments demonstrate that GeoNorm consistently outperforms existing normalization methods in Transformer models. Crucially, GeoNorm can be seamlessly integrated into standard Transformer architectures, achieving performance improvements with negligible additional computational cost.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
