发表机构
Leading University; DeepNet Research and Development Lab（莱丁大学; 深网研发实验室）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出受最速降线启发的学习率调度BrachistoneLR，通过受控基准实验证明平滑调度优于恒定和衰减策略，且策略形状比参数化更重要。
AI 中文摘要
学习率调度策略是训练深度网络时一个具有重要影响的选择，然而常用的策略多为启发式的，且已发表的比较结果难以解读，因为架构、数据集和预算往往随调度策略一同变化。我们研究了BrachistoneLR，这是一种通过将最速降线（重力作用下下降最快的曲线）的垂直坐标映射到峰值速率与下限速率之间的范围而构建的调度策略。扩展其定义可发现，它本质上是余弦退火，只是半周期设为E-1而非E，这正是标准实现在其周期参数比训练轮数少一时所采用的配置。因此，该速率在训练的最后一个轮次达到下限，而非晚一个轮次，我们证明这一差异随E^-2衰减，使其成为一种短视界效应。随后，我们在三个图像分类数据集（MNIST、Fashion-MNIST、CIFAR-10）和四个架构族（全连接、卷积、循环、残差）上，固定优化器、数据流水线和评估协议，仅改变调度策略，对六种调度策略进行了72次运行的基准测试。从峰值平滑下降至下限的调度策略优于恒定速率和基于日历的衰减策略，且优势随任务难度增加而扩大，在CIFAR-10上达到2.5个数据集的平均准确率百分点。在该领先组中，BrachistoneLR、余弦退火和带预热的余弦退火之间的准确率差异在0.06个百分点以内，平均排名差异在0.17以内，单一种子配置无法区分它们。BrachistoneLR在两种残差网络上均表现最佳，且在CIFAR-10上具有最高的平均准确率，并且它不需要设置里程碑、衰减因子、预热长度或重启周期。我们得出结论：调度策略的形状比其参数化更重要，是否使用平滑调度策略的选择比在平滑策略之间进行选择更重要，而终止速率差异仅在短视界内值得关注。
英文摘要
The learning-rate schedule is a consequential choice in training deep networks, yet the policies in common use are heuristic, and published comparisons are hard to read, because architecture, dataset, and budget tend to vary alongside the schedule. We study BrachistoneLR, a schedule built by mapping the vertical coordinate of the brachistochrone, the curve of fastest descent under gravity, onto the range between a peak and a floor rate. Expanding the definition shows it to be cosine annealing with the half-period set to E - 1 instead of E, the configuration a standard implementation gives when its period argument is one less than the number of epochs. The rate therefore reaches its floor at the last epoch trained rather than one epoch later, and we show this difference decays as E^-2, making it a short-horizon effect. We then benchmark six schedules over 72 runs on three image classification datasets (MNIST, Fashion-MNIST, CIFAR-10) and four architecture families (fully connected, convolutional, recurrent, residual), fixing the optimizer, data pipeline, and evaluation protocol so that only the schedule varies. Schedules that fall smoothly from peak to floor beat the constant rate and calendar-based decay by margins that grow with task difficulty, reaching 2.5 points of dataset mean on CIFAR-10. Within that leading group, BrachistoneLR, cosine annealing, and warmup-cosine lie within 0.06 accuracy points and 0.17 of a mean rank, which one seed per configuration cannot separate. BrachistoneLR is best on both residual networks and has the highest CIFAR-10 mean, and it sets no milestones, decay factor, warmup length, or restart period. We conclude that the shape of a schedule matters more than its parameterization, that the choice of whether to use a smooth schedule matters more than the choice among them, and that the terminal-rate distinction is worth attention only over short horizons.
