通过拥挤的山谷下降-基准深度学习优化器
Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
摘要：
选择优化器被认为是深度学习中最关键的设计决策之一，这并不容易。越来越多的文献现在列出了数百种优化方法。在缺乏明确的理论指导和确凿的经验证据的情况下，决策往往是基于轶事。在这项工作中，我们的目标是取代这些轶事，如果没有一个结论性的排名，那么至少有证据支持的证据。为此，我们对15种特别流行的深度学习优化器进行了广泛的标准化基准测试，同时简要概述了各种可能的选择。分析多个$50,000$单独运行，我们贡献了以下三点：（i）优化器性能在不同任务之间差异很大。(ii)我们观察到，使用默认参数评估多个优化器的效果与调整单个固定优化器的超参数大致相同。(iii)虽然我们无法辨别出在所有测试任务中明显占主导地位的优化方法，但我们确定了一个显着减少的特定优化器和参数选择的子集，这些优化器和参数选择通常会在我们的实验中产生竞争性结果：亚当仍然是一个强有力的竞争者，新的方法未能显著和持续地优于它。我们的开源结果具有挑战性，调整基线，对新型优化方法进行更有意义的评估，而无需任何进一步的计算工作。
英文摘要：
Choosing the optimizer is considered to be among the most crucial design decisions in deep learning, and it is not an easy one. The growing literature now lists hundreds of optimization methods. In the absence of clear theoretical guidance and conclusive empirical evidence, the decision is often made based on anecdotes. In this work, we aim to replace these anecdotes, if not with a conclusive ranking, then at least with evidence-backed heuristics. To do so, we perform an extensive, standardized benchmark of fifteen particularly popular deep learning optimizers while giving a concise overview of the wide range of possible choices. Analyzing more than $50,000$ individual runs, we contribute the following three points: (i) Optimizer performance varies greatly across tasks. (ii) We observe that evaluating multiple optimizers with default parameters works approximately as well as tuning the hyperparameters of a single, fixed optimizer. (iii) While we cannot discern an optimization method clearly dominating across all tested tasks, we identify a significantly reduced subset of specific optimizers and parameter choices that generally lead to competitive results in our experiments: Adam remains a strong contender, with newer methods failing to significantly and consistently outperform it. Our open-sourced results are available as challenging and well-tuned baselines for more meaningful evaluations of novel optimization methods without requiring any further computational efforts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
