全方位遮蔽梯度下降：通过遮蔽遍历实现内存高效的优化
Omni-Masked Gradient Descent: Memory-Efficient Optimization via Mask Traversal with Improved Convergence
AI总结：
Omni-Masked Gradient Descent通过遮蔽遍历实现内存高效的优化，提供非凸情况下改进的收敛分析和更优的迭代复杂度。
摘要：
为了解决GPU内存瓶颈下大语言模型全参数训练的扩展问题，近年来内存高效优化方法受到了越来越多的关注。现有方法要么缺乏清晰的收敛保证，要么只能在非凸情况下实现标准的O(ε^{-4})次迭代复杂度。我们提出了基于遮蔽遍历的Omni-Masked Gradient Descent（OMGD）优化方法，用于实现内存高效的训练，并提供了非凸收敛分析，建立了严格改进的迭代复杂度，即~O(ε^{-3})，用于找到ε-近似驻点。实证表明，OMGD是一种轻量级、即插即用的方法，能够无缝集成到大多数主流优化器中，在微调和预训练任务中均能一致地优于竞争基线。
英文摘要：
Memory-efficient optimization methods have recently gained increasing attention for scaling full-parameter training of large language models under the GPU-memory bottleneck. Existing approaches either lack clear convergence guarantees, or only achieve the standard ${\mathcal{O}}(ε^{-4})$ iteration complexity in the nonconvex settings. We propose Omni-Masked Gradient Descent (OMGD), an optimization method based on mask traversal for memory efficient training, and provide a nonconvex convergence analysis that establishes a strictly improved iteration complexity of $\tilde{\mathcal{O}}(ε^{-3})$ for finding an $ε$-approximate stationary point. Empirically, OMGD is a lightweight, plug-and-play approach that integrates seamlessly into most mainstream optimizers, yielding consistent improvements over competitive baselines in both fine-tuning and pre-training tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
