架构如何影响预训练语言模型的基础能力？基于FFN-Wider与MoE Transformer的案例研究
How does Architecture Influence the Base Capabilities of Pre-trained Language Models? A Case Study Based on FFN-Wider and MoE Transformers
AI总结：
本研究探究架构对预训练语言模型基础能力的影响，发现多头注意力贡献占比是关键因素，提出CEA解决FFN-Wider Transformer能力下降问题，并拓展至14B参数MoE模型实现显著提升，为架构设计提供指导。
摘要：
预训练语言模型已被证实具备强大的基础能力，不仅在分布内语言建模任务中表现优异，还在分布外语言建模、迁移学习和小样本学习中展现出突出能力。与现有聚焦规模对基础能力影响的研究不同，本工作探究架构对这些能力的影响，具体关注问题为：架构如何影响预训练语言模型的基础能力？本研究尝试解释并逆转FFN-Wider Transformer架构导致的基础能力下降，以期提供相关洞见。经分析发现，多头注意力（一种组合函数）对预训练语言建模的贡献占比是影响基础能力的关键因素；FFN-Wider Transformer降低了该组合函数的贡献占比，进而导致基础能力下降。我们通过实验验证了这一点，并提出组合增强架构（Combination Enhanced Architecture，CEA）以解决此类模型的基础能力下降问题。值得注意的是，我们将上述解释与CEA拓展至混合专家（Mixture of Experts，MoE）Transformer，在14B参数的MoE模型上成功实现了基础能力的显著提升，证明了本工作的实际应用价值，也表明我们的分析对架构分析、架构改进与架构设计具有一定指导意义。
英文摘要：
Pre-trained language models have been proven to possess strong base capabilities, which not only excel in in-distribution language modeling but also show powerful abilities in out-of-distribution language modeling, transfer learning and few-shot learning. Unlike existing work focusing on the influence of scale on base capabilities, our work examines the influence of architecture on those. Specifically, our concern is: How does architecture influence the base capabilities of pre-trained language models? In this work, we attempt to explain and reverse the decline in base capabilities caused by the architecture of FFN-Wider Transformers, seeking to provide some insights. Through analysis, we found the contribution ratio of Multi-Head Attention (a combination function) to pre-trained language modeling is a key factor affecting base capabilities. FFN-Wider Transformers reduce the contribution ratio of this combination function, leading to a decline in base capabilities. We confirmed this by experiments and proposed Combination Enhanced Architecture (CEA) to address the decline in base capabilities of such models. Significantly, we extended our explanation and CEA to Mixture of Experts (MoE) Transformers. We successfully achieved significant improvements in base capabilities on a 14B parameter MoE model, demonstrating the practical application value of our work. This also indicates that our analysis has a certain guiding significance for architecture analysis, architecture improvement and architecture design.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
