Pruning eliminates unnecessary parameters in neural networks; it offers a promising solution to the growing computational demands of large language models (LLMs). While many focus on post-training pruning, sparse pre-training--which combines pruning and pre-training into a single phase--provides a simpler alternative. In this work, we present the first systematic exploration of optimal sparse pre-training configurations for LLMs through an examination of 80 unique pruning schedules across different sparsity levels and training durations. We find that initiating pruning at 25% of total training compute and concluding at 75% achieves near-optimal final evaluation loss. These findings provide valuable insights for efficient and effective sparse pre-training of LLMs. Furthermore, we propose a new scaling law that modifies the Chinchilla scaling law to use the average parameter count over pre-training. Through empirical and theoretical validation, we demonstrate that this modified scaling law accurately models evaluation loss for both sparsely and densely pre-trained LLMs, unifying scaling laws across pre-training paradigms. Our findings indicate that while sparse pre-training achieves the same final model quality as dense pre-training for equivalent compute budgets, it provides substantial benefits through reduced model size, enabling significant potential computational savings during inference.
METEOR: Evolutionary Journey of Large Language Models from Guidance to Self-Growth
School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；Beijing University of Posts and Telecommunications（北京邮电大学）
Task-Stratified Knowledge Scaling Laws for Post-Training Quantized Large Language Models
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学先进交叉学科学院）；The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所认知与决策智能复杂系统重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；College of Computer Science, Inner Mongolia University（内蒙古大学计算机学院）
A Unified Generalization Framework for Model Merging: Trade-offs, Non-Linearity, and Scaling Laws
National University of Defense Technology（国防科技大学）；Wuhan University（武汉大学）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学信息科学与技术学院深圳校区）；Hebei University of Technology（河北工业大学）
LAWS: Look Around and Warm-Start Natural Gradient Descent for Quantum Neural Networks
William & Mary（威廉与玛丽学院）
变分量子算法（VQA）最近受到了研究界的极大关注，因为它们在有噪声的中间尺度量子计算机（NISQ）中有很好的性能。然而，在具有随机初始化参数的参数化量子线路（PQC）上运行的VQA的特征在于贫瘠高原（BP），其中梯度在量子比特的数量上呈指数地消失。在本文中，我们首先回顾量子自然梯度（QNG），这是最流行的算法之一，在VQA，从经典的一阶优化的角度来看。然后，我们提出了一个\underline{L} ook \underline{A} round \underline{W} arm- \underline{S} tart QNG（LAWS）算法，以减轻广泛存在的BP问题。LAWS是一种组合优化策略，它利用了模型参数初始化和QNG快速收敛的优点。LAWS重复地重新调整参数搜索空间，以用于下一次迭代参数更新。重新初始化的参数搜索空间是通过对接近当前最优的梯度进行采样来仔细选择的。此外，我们提出了一个统一的框架（WS-SGD）集成参数初始化技术到优化器。我们提供了收敛性证明的建议框架的凸和非凸目标函数的基础上Polyak-Lojasiewicz（PL）条件。实验结果表明，该算法在量子分类问题中能有效地降低BP算法的泛化能力。
Effective Frontiers: A Unification of Neural Scaling Laws
Department of Mathematics and Statistics, Xi'an Jiaotong University（西安交通大学数学与统计学系）；Department of Artificial Intelligence, Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）；University of Chinese Academy of Sciences（中国科学院大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
