Large language models are increasingly trained in continual or open-ended settings, where the total training horizon is not known in advance. Despite this, most existing pretraining recipes are not anytime: they rely on horizon-dependent learning rate schedules and extensive tuning under a fixed compute budget. In this work, we provide a theoretical analysis demonstrating the existence of anytime learning schedules for overparameterized linear regression, and we highlight the central role of weight averaging - also known as model merging - in achieving the minimax convergence rates of stochastic gradient descent. We show that these anytime schedules polynomially decay with time, with the decay rate determined by the source and capacity conditions of the problem. Empirically, we evaluate 150M and 300M parameter language models trained for up to 32x Chinchilla scale, comparing constant and $1/\sqrt{t}$ schedules with weight averaging against a well-tuned cosine schedule. Across the full training range, the anytime schedules achieve comparable final loss to cosine decay. Taken together, our results suggest that weight averaging combined with simple, horizon-free step sizes offers a practical and effective anytime alternative to cosine learning rate schedules for large language model pretraining.
Taking the Road Less Scheduled with Adaptive Polyak Steps
Johns Hopkins University（约翰霍普金斯大学）；University of British Columbia（不列颠哥伦比亚大学）；Australian National University（澳大利亚国立大学）；Center for Computational Mathematics（计算数学中心）；Flatiron Institute, Simons Foundation（Flatiron 机构，Simons 基金会）
Experiments on Parallel Training of Deep Neural Network using Model Averaging
International Computer Science Institute（国际计算机科学研究所）；University of California, Berkeley（加州大学伯克利分校）；Nanyang Technological University（南洋理工大学）
在这项工作中，我们将模型平均应用于深度神经网络（DNN）的并行训练。以模型平均化的方式进行模型化。数据被分区并分发到不同的节点以进行本地模型更新，并且每隔几个小批进行跨节点的模型平均。我们使用多个GPU进行数据并行化，并使用消息传递接口（MPI）进行节点之间的通信，这使我们能够频繁地执行模型平均，而不会在通信上浪费太多时间。研究了自然梯度随机梯度下降（NG-SGD）和受限玻尔兹曼机（RBM）预训练在模型平均框架下并行训练的有效性，并根据不同的学习率调度、平均频率和小批量大小探索了最佳设置。结果表明，NG-SGD和RBM预训练有利于基于参数平均的模型训练。在300 h Switchboard数据集上，使用16个GPU实现了9.3倍的加速，使用32个GPU实现了17倍的加速，解码精度损失有限。
Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss
Eindhoven University of Technology（埃因霍温理工大学）；Georgia Institute of Technology（佐治亚理工学院）；University of Freiburg（弗赖堡大学）；LAION；Open-Ψ (Open-Sci) Collective（Open-Ψ（开放科学）集体）；University of Turku（图尔库大学）；OpenEuroLLM
Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Models
School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism
Perelman School of Medicine, University of Pennsylvania（宾夕法尼亚大学佩雷尔曼医学院）；The University of Chicago Booth School of Business（芝加哥大学布斯商学院）；Northwestern University（西北大学）；University of Southern California Marshall School of Business（南加州大学马歇尔商学院）
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shenzhen University of Advanced Technology（深圳大学）；Pengcheng Laboratory（鹏城实验室）；University of Chinese Academy of Sciences（中国科学院大学）
Towards Understanding Momentum Acceleration in River-Valley Loss Landscape
Stanford University（斯坦福大学）；University of California, San Diego（加州大学圣迭戈分校）；University of Chicago（芝加哥大学）；Yale University（耶鲁大学）；Toyota Technological Institute at Chicago（芝加哥丰田技术学院）
Differentially Private Zeroth-Order Methods for Scalable Large Language Model Finetuning
Zhejiang University（浙江大学）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新区（滨江）区块链与数据安全研究院）；University of Chicago（芝加哥大学）
IQuest Research（IQuest研究公司）；Peking University（北京大学）；Sun Yat-sen University（中山大学）；Shenzhen University of Advanced Technology（深圳先进技术大学）；Shanghai University of Finance and Economics（上海财经大学）
