arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.16690cs.CL
收藏

线性复杂度语言模型的缩放定律

Scaling Laws for Linear Complexity Language Models

OpenNLPLab(开放自然语言处理实验室);Australian National University(澳大利亚国立大学);TapTap

Xuyang Shen, Dong Li, Ruitao Leng, Zhen Qin, Weigao Sun, Yiran Zhong

AI总结:

本研究探究线性复杂度语言模型的缩放定律,对比三种线性架构与LLaMA基线在多任务上的表现,发现其缩放能力与传统Transformer模型相当,且语言能力和知识保留更优。

摘要:

尽管大型语言模型的线性复杂度模型缩放能力仍不明确,但人们对其的兴趣正不断上升。本研究提出线性复杂度语言模型的缩放定律,为其可扩展性奠定基础。具体而言,我们研究了三种高效线性架构的缩放行为:包括TNL(一种具有数据无关衰减的线性注意力模型)、HGRN2(一种具有数据相关衰减的线性循环神经网络)以及cosFormer2(一种无衰减的线性注意力模型)。我们还纳入了采用softmax注意力的基线架构LLaMA用于对比。这些模型有6种不同规模变体,参数范围从70M到7B,在300B token的语料库上训练,并使用总共1376个中间检查点在各类下游任务上评估。这些任务包括验证损失、常识推理以及信息检索与生成。研究表明,现有线性复杂度语言模型展现出与传统基于Transformer的模型相近的缩放能力,同时还表现出更优的语言能力和知识保留能力。

英文摘要:

The interest in linear complexity models for large language models is on the rise, although their scaling capacity remains uncertain. In this study, we present the scaling laws for linear complexity language models to establish a foundation for their scalability. Specifically, we examine the scaling behaviors of three efficient linear architectures. These include TNL, a linear attention model with data-independent decay; HGRN2, a linear RNN with data-dependent decay; and cosFormer2, a linear attention model without decay. We also include LLaMA as a baseline architecture for softmax attention for comparison. These models were trained with six variants, ranging from 70M to 7B parameters on a 300B-token corpus, and evaluated with a total of 1,376 intermediate checkpoints on various downstream tasks. These tasks include validation loss, commonsense reasoning, and information retrieval and generation. The study reveals that existing linear complexity language models exhibit similar scaling capabilities as conventional transformer-based models while also demonstrating superior linguistic proficiency and knowledge retention.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑