arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.13442cs.LGcs.AIcs.CLstat.ML
收藏

大语言模型中的下一词元预测定律

A Law of Next-Token Prediction in Large Language Models

University of Rochester(罗切斯特大学);University of Pennsylvania(宾夕法尼亚大学)

Hangfeng He, Weijie J. Su

AI总结:

该研究提出大语言模型下一词元预测的定量定律,发现各层对预测准确率提升贡献均等且跨架构通用,可为模型开发与应用提供新指导。

摘要:

大语言模型(LLMs)已被广泛应用于各类领域,但其黑箱特性为理解模型内部如何处理输入数据并生成预测带来了重大挑战。本文提出了一条精确的定量定律,用于描述预训练LLMs中间层在执行下一词元预测任务时,上下文化词元嵌入的学习规律。研究发现,从最底层到最高层,每一层对提升预测准确率的贡献均等——这是一种在多种开源LLMs中普遍存在的现象,与模型架构或预训练数据无关。我们证明,该定律能为LLM的开发与应用实践(包括模型缩放、预训练任务和可解释性研究)提供新的视角与可落地的指导见解。

英文摘要:

Large language models (LLMs) have been widely employed across various application domains, yet their black-box nature poses significant challenges to understanding how these models process input data internally to make predictions. In this paper, we introduce a precise and quantitative law that governs the learning of contextualized token embeddings through intermediate layers in pre-trained LLMs for next-token prediction. Our findings reveal that each layer contributes equally to enhancing prediction accuracy, from the lowest to the highest layer -- a universal phenomenon observed across a diverse array of open-source LLMs, irrespective of their architectures or pre-training data. We demonstrate that this law offers new perspectives and actionable insights to inform and guide practices in LLM development and applications, including model scaling, pre-training tasks, and interpretation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑