arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.04066cs.CLcs.AI
收藏

超越规模:测量和预测语言模型预训练中知识保留的上限

Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Training

Fudan NLP Group(复旦大学自然语言处理组);IEIT Systems Co. Ltd.(浪潮信息系统有限公司)

Changhao Jiang, Ming Zhang, Yifei Cao, Junjie Ye, Xiaoran Fan, Shihan Dou, Zhiheng Xi, Jiajun Sun, Yi Dong, Yujiong Shen, Jingqi Tong, Baoyu Fan, Tao Gui, Qi Zhang, Xuanjing Huang

AI总结:

本文提出SMI方法,通过整合知识频率、特异性与模型大小,预测语言模型在封闭书目问答中的准确性,揭示预训练知识保留的上限。

摘要:

GPT-4技术报告指出,下游性能可以从预训练信号中预测,但并未提供如何量化这一关系的方法学细节。本文通过建模知识保留能力,即预训练语言模型记忆其语料库中事实信息的能力,提出了一种系统的方法来在训练前估计这一能力。我们提出了大小依赖互信息(SMI),一种信息论预测器,它整合了知识频率、知识特异性以及模型大小,以预测封闭书目问答(QA)的准确性。SMI通过在披露的预训练语料库上进行大规模文档检索,结合稳健的多模板QA评估进行验证。实验显示,SMI在预测模型超过10亿参数时的QA准确性方面显著优于基于重复的基线方法,并且在无额外训练的情况下实现了R平方大于0.7。分析进一步揭示了数据和模型规模扩展的边际收益递减,并提供了通过预训练单独实现的知识保留的内在上限的证据,这推动了检索和其他增强策略的发展。数据集和代码可在https://github.com/yuhui1038/SMI上获得。

英文摘要:

The GPT-4 technical report suggests that downstream performance can be predicted from pre-training signals, but offers little methodological detail on how to quantify this. This work address this gap by modeling knowledge retention, the capacity of a pre-trained language model to memorize factual information from its corpus, and introduce a principled method to estimate it prior to training. We propose Size-dependent Mutual Information (SMI), an information-theoretic predictor that integrates knowledge frequency, knowledge specificity, and model size to forecast closed-book question answering (QA) accuracy. SMI is validated through large-scale document retrieval over the disclosed pre-training corpora of 21 public and 3 custom models, combined with a robust multi-template QA evaluation. Experiments show that SMI significantly outperforms repetition-based baselines and achieves $R^2$ > 0.7 in predicting QA accuracy for models above 1B parameters, without additional training. The analysis further reveals diminishing returns from scaling data and model size and provides evidence for an intrinsic upper bound on knowledge retention achievable by pre-training alone, motivating retrieval and other augmentation strategies. The dataset and code are available at https://github.com/yuhui1038/SMI.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑