arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.04848cs.CL
收藏

大语言模型预训练过程中下游任务性能的不稳定性

Instability in Downstream Task Performance During LLM Pretraining

Nara Institute of Science and Technology(那拉科学与技术研究所);Tohoku University(东北大学);Research and Development Center for Large Language Models, National Institute of Informatics(大型语言模型研究与开发中心,信息学国家研究所)

Yuto Nishida, Masaru Isonuma, Yusuke Oda

AI总结:

本研究分析LLM预训练中下游任务性能的波动性,提出检查点平均与集成两种事后方法,在不改变训练过程的情况下有效提升性能稳定性。

摘要:

在训练大型语言模型(LLM)时,常见的做法是在整个训练过程中跟踪下游任务性能,并选择验证分数最高的检查点。然而,下游指标往往表现出大幅波动,使得难以确定真正代表最佳性能模型的检查点。在本研究中,我们实证分析了在多样化网络规模语料库上训练的LLM的下游任务性能的稳定性。我们发现,任务分数在整个训练过程中频繁波动,无论是在总体层面还是示例层面。为了解决这种不稳定性,我们研究了两种事后检查点集成方法:检查点平均和集成,其动机是假设聚合相邻检查点可以减少性能波动。我们在实证和理论上都证明了这些方法在不改变训练过程的情况下提高了下游性能的稳定性。

英文摘要:

When training large language models (LLMs), it is common practice to track downstream task performance throughout the training process and select the checkpoint with the highest validation score. However, downstream metrics often exhibit substantial fluctuations, making it difficult to identify the checkpoint that truly represents the best-performing model. In this study, we empirically analyze the stability of downstream task performance in an LLM trained on diverse web-scale corpora. We find that task scores frequently fluctuate throughout training, both at the aggregate and example levels. To address this instability, we investigate two post-hoc checkpoint integration methods: checkpoint averaging and ensemble, motivated by the hypothesis that aggregating neighboring checkpoints can reduce performance volatility. We demonstrate both empirically and theoretically that these methods improve downstream performance stability without requiring any changes to the training procedure.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑