AOS：基于训练状态信号的自适应优化器切换，实现更快收敛与更好泛化
AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization
浏览论文内容
中文总结 AI 辅助
AOS-R通过监测6种梯度空间信号切换AdamW等优化器，在8组基准测试中6组获最佳准确率，收敛速度较AdamW提升0.80倍且精度平均提高0.4个百分点。
中文摘要 AI 辅助
单一优化器训练不适用于深度网络优化的不同阶段：自适应方法能很好地处理早期含噪声的梯度，但会在平坦极小值处过冲；带动量的SGD（SGD-M）在后期阶段泛化能力更好，但早期收敛缓慢。我们提出AOS-R（自适应优化器切换，基于规则），这是一种轻量型控制器，它监测6种在线梯度空间信号——梯度噪声尺度（GNS）、Hutchinson曲率迹、损失停滞、更新稳定比、梯度稳定指数（GSI）以及损失改善率（LIR）——并根据优化场景的演变在AdamW、SGD-M和Lion之间切换。状态保持动量转移与400步学习率过渡避免了每次切换点的精度下降。在CIFAR-100/WRN-28x10上，AOS-R在81个epoch达到78%的Top-1准确率，比AdamW（109个epoch）少26%，比SGD-M（143个epoch）少43%，比Lion（96个epoch）少16%。在8组模型-数据集基准测试中，AOS-R在6组组合上取得最佳准确率，在单一共享超参数配置下，相比AdamW平均精度提升0.4个百分点，收敛速度加快0.80倍。
英文摘要
Single-optimizer training is a poor fit for the distinct phases of deep network optimization: adaptive methods handle noisy early gradients well but overshoot flat minima, while SGD with momentum generalizes better in the late phase but converges slowly early on. We introduce AOS-R (Adaptive Optimizer Switching, Rule-Based), a lightweight controller that monitors six online gradient-space signals -- gradient noise scale (GNS), Hutchinson curvature trace, loss stagnation, update stability ratio, gradient stability index (GSI), and loss improvement ratio (LIR) -- and switches among AdamW, SGD-M, and Lion as the optimization landscape evolves. State-preserving momentum transfer and a 400-step learning-rate bridge prevent accuracy degradation at every transition point. On CIFAR-100/WRN-28x10, AOS-R reaches 78% top-1 in 81 epochs -- 26% fewer than AdamW (109), 43% fewer than SGD-M (143), and 16% fewer than Lion (96). Across eight model-dataset benchmarks, AOS-R achieves best accuracy on 6 of 8 combinations with a mean +0.4 pp gain and 0.80x convergence speedup over AdamW under a single shared hyperparameter configuration.
