DenseFormer：通过深度加权平均增强 Transformer 中的信息流
DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging
AI总结：
本文提出 DenseFormer 架构，通过在每个 transformer 块后引入 Depth-Weighted-Average (DWA) 操作计算当前与过去表示的加权平均，在不显著增加参数量的前提下提升了模型困惑度、数据效率及推理性能。
摘要：
Vaswani 等人（2017）提出的 transformer 架构目前在从自然语言处理到语音处理和图像理解等应用领域中无处不在。我们提出了 DenseFormer，这是对标准架构的一项简单修改，它在不增加模型规模的情况下改善了模型的困惑度——对于 100B 参数范围内的大规模模型仅增加几千个参数。我们的方法依赖于在每个 transformer 块之后增加一个额外的平均步骤，该步骤计算当前和过去表示的加权平均值——我们将此操作称为 Depth-Weighted-Average (DWA)。学习到的 DWA 权重展现出连贯的信息流模式，揭示了对来自远处层激活的强烈且结构化的重用。实验表明，DenseFormer 具有更高的数据效率，能够达到与深得多的 transformer 模型相同的困惑度，并且在相同困惑度下，这些新模型在内存效率和推理时间方面优于 transformer 基线。
英文摘要：
The transformer architecture by Vaswani et al. (2017) is now ubiquitous across application domains, from natural language processing to speech processing and image understanding. We propose DenseFormer, a simple modification to the standard architecture that improves the perplexity of the model without increasing its size -- adding a few thousand parameters for large-scale models in the 100B parameters range. Our approach relies on an additional averaging step after each transformer block, which computes a weighted average of current and past representations -- we refer to this operation as Depth-Weighted-Average (DWA). The learned DWA weights exhibit coherent patterns of information flow, revealing the strong and structured reuse of activations from distant layers. Experiments demonstrate that DenseFormer is more data efficient, reaching the same perplexity of much deeper transformer models, and that for the same perplexity, these new models outperform transformer baselines in terms of memory efficiency and inference time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
