arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.18158stat.MLcs.LG
收藏

解锁 Token 作为数据点以推导更大语言模型的泛化界限

Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Models

New York University(纽约大学);Meta AI;Carnegie Mellon University(卡内基梅隆大学)

Sanae Lotfi, Yilun Kuang, Brandon Amos, Micah Goldblum, Marc Finzi, Andrew Gordon Wilson

AI总结:

本文利用鞅性质推导基于海量 token 的泛化界限,结合 Monarch 矩阵、Kronecker 分解和训练后量化,首次为 LLaMA2-70B 等实际部署且生成高质量文本的大模型实现了非空泛化界限。

摘要:

具有数十亿参数的大型语言模型(LLM)在预测序列中的下一个 token 方面表现出色。近期的研究为 LLM 计算了非空的基于压缩的泛化界限,但这些界限对于十亿参数规模的大模型而言是空的。此外,这些界限是通过限制性的压缩技术获得的,限制了生成低质量文本的压缩模型。另外,这些现有界限的紧度取决于训练集中 IID 文档的数量,而不是数量大得多的非 IID 组成 token,这使得更紧界限的潜力未被开发。在这项工作中,我们转而利用鞅的性质推导泛化界限,这些界限得益于 LLM 训练集中海量的 token。由于数据集包含的 token 远多于文档,我们的泛化界限不仅容忍,而且实际上受益于限制性远低的压缩方案。借助 Monarch 矩阵、Kronecker 分解和训练后量化,我们为高达 LLaMA2-70B 的 LLM 实现了非空的泛化界限。与以往方法不同,我们的工作首次为实际部署且能生成高质量文本的模型实现了非空界限。

英文摘要:

Large language models (LLMs) with billions of parameters excel at predicting the next token in a sequence. Recent work computes non-vacuous compression-based generalization bounds for LLMs, but these bounds are vacuous for large models at the billion-parameter scale. Moreover, these bounds are obtained through restrictive compression techniques, bounding compressed models that generate low-quality text. Additionally, the tightness of these existing bounds depends on the number of IID documents in a training set rather than the much larger number of non-IID constituent tokens, leaving untapped potential for tighter bounds. In this work, we instead use properties of martingales to derive generalization bounds that benefit from the vast number of tokens in LLM training sets. Since a dataset contains far more tokens than documents, our generalization bounds not only tolerate but actually benefit from far less restrictive compression schemes. With Monarch matrices, Kronecker factorizations, and post-training quantization, we achieve non-vacuous generalization bounds for LLMs as large as LLaMA2-70B. Unlike previous approaches, our work achieves the first non-vacuous bounds for models that are deployed in practice and generate high-quality text.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑