arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.20280cs.CLcs.AI
收藏

通过学习预测上下文嵌入进行上下文级语言建模

Context-level Language Modeling by Learning Predictive Context Embeddings

LUMIA Lab(LUMIA实验室);School of Artificial Intelligence(人工智能学院);Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Tsinghua University(清华大学);Nanjing University(南京大学)

Beiya Dai, Yuliang Liu, Daozheng Xue, Yunchong Song, Qipeng Guo, Kai Chen, Xinbing Wang, Bowen Zhou, Zhouhan Lin

AI总结:

ContextLM通过学习预测上下文嵌入,提升多令牌预测能力,实现更高效的参数和训练效率,同时在下游任务中表现更优。

摘要:

我们提出了ContextLM,一个框架,通过在标准预训练中加入内在的下一步上下文预测目标,隐式地学习多令牌预测。ContextLM在跨越多个令牌的上下文嵌入之上构建语言模型,通过预测下一个上下文来实现更好的下一个令牌预测。我们的模型完全兼容标准的自回归、逐令牌评估范式(例如,困惑度)。使用GPT-2和Pythia架构(最多15亿参数和3000亿训练令牌)的大量实验表明,ContextLM改变了缩放定律的帕累托前沿,表现出在参数、训练令牌和FLOPs方面的优越效率。我们的结果表明,ContextLM在39%更少的参数下已经能够达到基线困惑度,并在等效参数数量下在广泛的下游任务上展示了稳健的泛化改进。

英文摘要:

We propose ContextLM, a framework that implicitly learns multi-token prediction by augmenting standard pretraining with an intrinsic next-context prediction objective. ContextLM builds a language model on top of context embeddings that span multiple tokens, enabling better next-token prediction by predicting the next context. Our model is fully compatible with standard autoregressive, token-by-token evaluation paradigms (e.g., perplexity). Extensive experiments with GPT-2 and Pythia backbones (up to 1.5B parameters and 300B training tokens) reveal that ContextLM shifts the Pareto frontier of scaling laws, exhibiting superior efficiency in parameters, training tokens, and FLOPs. Our results show that ContextLM could already achieve the baseline perplexity using 39\% fewer parameters and demonstrates robust generalization improvements on extensive downstream tasks under equivalent parameter counts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑