SPARKLING：平衡信号保留与对称性破坏以实现宽度渐进学习
SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning
AI总结：
SPARKLING通过信号保留和对称性破坏技术，实现中间阶段宽度扩展，提升模型训练效率并减少计算成本。
摘要：
SPARKLING通过信号保留和对称性破坏技术，实现中间阶段宽度扩展，提升模型训练效率并减少计算成本。
英文摘要：
Progressive Learning (PL) reduces pre-training computational overhead by gradually increasing model scale. While prior work has extensively explored depth expansion, width expansion remains significantly understudied, with the few existing methods limited to the early stages of training. However, expanding width during the mid-stage is essential for maximizing computational savings, yet it remains a formidable challenge due to severe training instabilities. Empirically, we show that naive initialization at this stage disrupts activation statistics, triggering loss spikes, while copy-based initialization introduces gradient symmetry that hinders feature diversity. To address these issues, we propose SPARKLING (balancing {S}ignal {P}reservation {A}nd symmet{R}y brea{K}ing for width-progressive {L}earn{ING}), a novel framework for mid-stage width expansion. Our method achieves signal preservation via RMS-scale consistency, stabilizing activation statistics during expansion. Symmetry breaking is ensured through asymmetric optimizer state reset and asymmetric learning rate re-warmup. Extensive experiments on dense and Mixture-of-Experts (MoE) models demonstrate that, across multiple width axes and optimizer families, SPARKLING consistently outperforms training from scratch and reduces training cost by up to 35% under $2\times$ width expansion.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
