arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.14549cs.CLcs.AI
收藏

双目标语言模型:无需过拟合的训练效率

Dual-objective Language Models: Training Efficiency Without Overfitting

University of Oslo(奥斯陆大学);National Library of Norway(挪威国家图书馆)

David Samuel, Lucas Georges Gabriel Charpentier

AI总结:

本文结合自回归和掩码扩散训练目标,提出灵活的语言模型,优于单一目标模型。通过训练50个模型,证明双目标训练在效率与抗过拟合间取得最佳平衡。

摘要:

本文结合自回归和掩码扩散训练目标,无需架构修改,实现灵活的语言模型,优于单一目标模型。自回归建模因训练效率高而流行,但易受过拟合影响。而掩码扩散模型训练效率较低但抗过拟合更强。本文证明双目标训练兼顾两者优点。通过训练和评估50个模型,在不同数据重复水平下,证明双目标结合在所有评估设置中最优,且最佳平衡在追求自回归或掩码扩散下游性能时相似。

英文摘要:

This paper combines autoregressive and masked-diffusion training objectives without any architectural modifications, resulting in flexible language models that outperform single-objective models. Autoregressive modeling has been a popular approach, partly because of its training efficiency; however, that comes at the cost of sensitivity to overfitting. On the other hand, masked-diffusion models are less efficient to train while being more resilient to overfitting. In this work, we demonstrate that dual-objective training achieves the best of both worlds. To derive the optimal balance between both objectives, we train and evaluate 50 language models under varying levels of data repetition. We show that it is optimal to combine both objectives under all evaluated settings and that the optimal balance is similar whether targeting autoregressive or masked-diffusion downstream performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑