arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.16450cs.CL
收藏

构建高效基础:有效训练具有结构化前馈层的大型语言模型

Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers

EPFL(洛桑联邦理工学院);Google DeepMind(谷歌DeepMind)

Xiuying Wei, Skander Moalla, Razvan Pascanu, Caglar Gulcehre

AI总结:

本研究针对Transformer LLM中计算密集的前馈网络,采用低秩和块对角结构化参数化,从零训练至1.3B参数,提出自引导训练改善训练动态,证明结构化网络在扩展FLOPs和过度训练中更高效,以更少参数和更低损失优于密集模型。

摘要:

大型语言模型(LLMs)的最先进结果往往依赖于规模,这导致计算成本高昂。这引发了旨在减少这些模型参数数量和计算成本而不显著影响其性能的研究议程。我们的研究聚焦于基于Transformer的LLMs,特别针对计算密集的前馈网络(FFNs),这些网络相比注意力模块研究较少。我们考虑使用高效低秩和块对角矩阵对FFN进行三种结构化线性参数化。与许多先前研究这些近似的工作不同,我们的研究i)从从头训练的角度探索这些结构,ii)扩展到1.3B参数,iii)在最近的基于Transformer的LLMs而非卷积架构中进行。我们证明这些结构在各种场景中可以带来实际的计算收益,包括使用预合并技术的在线解码。此外,我们提出了一种新颖的训练机制,称为自引导训练,旨在改善这些近似从初始化使用时表现出的不良训练动态。有趣的是,我们探索了结构化矩阵的扩展性能,揭示了在扩展训练FLOPs时更陡峭的曲线,以及在过度训练机制中有利的扩展趋势。具体来说,我们表明宽且结构化的网络可以更有效地利用训练FLOPs,在最优权衡下以更少的参数和更低的损失优于密集模型。我们的代码可在\url{https://github.com/CLAIRE-Labo/StructuredFFN/tree/main}获取。

英文摘要:

State-of-the-art results in large language models (LLMs) often rely on scale, which becomes computationally expensive. This has sparked a research agenda to reduce these models' parameter counts and computational costs without significantly impacting their performance. Our study focuses on transformer-based LLMs, specifically targeting the computationally intensive feedforward networks (FFNs), which are less studied than attention blocks. We consider three structured linear parameterizations of the FFN using efficient low-rank and block-diagonal matrices. In contrast to many previous works that examined these approximations, our study i) explores these structures from a training-from-scratch perspective, ii) scales up to 1.3B parameters, and iii) is conducted within recent Transformer-based LLMs rather than convolutional architectures. We demonstrate that these structures can lead to actual computational gains in various scenarios, including online decoding when using a pre-merge technique. Additionally, we propose a novel training regime, called \textit{self-guided training}, aimed at improving the poor training dynamics that these approximations exhibit when used from initialization. Interestingly, the scaling performance of structured matrices is explored, revealing steeper curves in scaling training FLOPs, along with a favorable scaling trend in the overtraining regime. Specifically, we show that wide and structured networks can utilize training FLOPs more efficiently, with fewer parameters and lower loss than dense models at their optimal trade-off. Our code is available at \url{https://github.com/CLAIRE-Labo/StructuredFFN/tree/main}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑