arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.01637cs.CLcs.LG
收藏

扩展语言模型中的嵌入层

Scaling Embedding Layers in Language Models

Google(谷歌)

Da Yu, Edith Cohen, Badih Ghazi, Yangsibo Huang, Pritish Kamath, Ravi Kumar, Daogao Liu, Chiyuan Zhang

AI总结:

本文提出SCONE方法,通过扩展n-gram嵌入层并卸载至加速器外,在保持推理效率的同时提升语言模型性能,使10亿参数模型超越19亿参数基线。

摘要:

我们提出了SCONE(可扩展、上下文相关、卸载、N-gram嵌入),一种用于扩展输入嵌入层以提升语言模型性能的新方法。为避免增加解码成本,SCONE在保留原始词汇表的同时,引入了一组频繁n-gram的嵌入。这些嵌入为每个输入标记提供上下文相关的表示,并在训练期间通过一个单独的模型进行学习。训练后,嵌入被预先计算并存储在加速器外部内存中;在推理期间,由于嵌入查找的复杂度低,查询它们对延迟的影响极小。SCONE支持两种新的扩展策略:增加n-gram嵌入的数量以及扩展用于学习它们的模型,同时保持推理期间固定的加速器使用量(以FLOPS和内存计)。我们表明,扩展这两个方面使得一个拥有10亿加速器驻留参数的模型在各种语料库上优于19亿参数的基线模型,同时推理期间仅使用约一半的FLOPS和加速器内存。

英文摘要:

We propose $SCONE$ ($S$calable, $C$ontextualized, $O$ffloaded, $N$-gram $E$mbedding), a new method for extending input embedding layers to enhance language model performance. To avoid increased decoding costs, $SCONE$ retains the original vocabulary while introducing embeddings for a set of frequent n-grams. These embeddings provide contextualized representation for each input token and are learned with a separate model during training. After training, embeddings are precomputed and stored in off-accelerator memory; during inference, querying them has minimal impact on latency due to the low complexity of embedding lookups. $SCONE$ enables two new scaling strategies: increasing the number of n-gram embeddings and scaling the model used to learn them, both while maintaining fixed accelerator usage during inference (in terms of FLOPS and memory). We show that scaling both aspects enables a model with 1B accelerator-resident parameters to outperform a 1.9B-parameter baseline across diverse corpora, while using only about half the FLOPS and accelerator memory during inference.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑