arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.14746cs.CLcs.LG
收藏

扩展高效大语言模型

Scaling Efficient LLMs

Independent（独立研究者）

B. N. Kausik

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出递归变换器，通过结合变换器和递归网络的优势，实现了高效的大语言模型，解决了参数规模与数据规模之间的非线性关系问题。

摘要：

最近的大语言模型（LLM）拥有数百亿个参数，消耗了大量资源。此外，所谓的

英文摘要：

Recent LLMs have hundreds of billions of parameters consuming vast resources. Furthermore, the so called "AI scaling law" for transformers suggests that the number of parameters must scale linearly with the size of the data. In response, we inquire into efficient LLMs, i.e. those with the fewest parameters that achieve the desired accuracy on a training corpus. Specifically, by comparing theoretical and empirical estimates of the Kullback-Leibler divergence, we derive a natural AI scaling law that the number of parameters in an efficient LLM scales as $D^γ$ where $D$ is the size of the training data and $ γ\in [0.44, 0.72]$, suggesting the existence of more efficient architectures. Against this backdrop, we propose recurrent transformers, combining the efficacy of transformers with the efficiency of recurrent networks, progressively applying a single transformer layer to a fixed-width sliding window across the input sequence. Recurrent transformers (a) run in linear time in the sequence length, (b) are memory-efficient and amenable to parallel processing in large batches, (c) learn to forget history for language tasks, or accumulate history for long range tasks like copy and selective copy, and (d) are amenable to curriculum training to overcome vanishing gradients. In our experiments, we find that recurrent transformers perform favorably on benchmark tests.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑