众多优化器但仅一条训练路径：用于自适应优化器选择的重复重采样
Many Optimizers But Only One Training Path: Repeated Resampling for Adaptive Optimizer Selection
查看机构详情
- insureAI
- ETH Zürich（苏黎世联邦理工学院）
机构由 AI 辅助整理，请以论文原文为准。
浏览论文内容
中文总结 AI 辅助
该研究提出重复优化器重采样（ROR）方法，在单次训练中搜索最优优化器，仅需穷举训练的24%-35%量，即可在多任务上接近最优固定优化器性能，实现高效自适应优化器选择。
中文摘要 AI 辅助
在训练深度神经网络前通常会选择一个优化器并保持其固定。将优化器选择视为超参数可提升性能，但需要多次完整训练运行，且除最优者外其余均被丢弃。重复优化器重采样（Repeated Optimizer Resampling，ROR）则在一次演化训练运行中进行搜索：每b个轮次，每个候选优化器从当前模型权重出发进行s个轮次的探索；最优探索结果将继续完成剩余b-s个轮次，若该完成的片段能提升验证目标，则成为新的当前最优优化器。该设计允许偏好的优化器随训练进程变化。我们在MNIST、Fashion-MNIST及两个汽车保险索赔计数模型上对比了ROR的两个变体，以相同的10个随机种子评估了9个固定优化器和两个ROR变体。单轮次ROR的总训练量仅为穷举识别最优固定优化器所需的24%至35%，且在所有四个任务上均接近该最优固定优化器。这些结果表明，短探索是一种实用的优化器搜索方式，无需完成所有候选运行即可实现优化器搜索。
英文摘要
An optimizer is usually chosen before training a deep neural network and then kept fixed. Treating optimizer choice as a hyperparameter could boost performance, but it requires several complete training runs and discards all but the winner. Repeated Optimizer Resampling (ROR) instead searches during one evolving run. Every $b$ epochs, each candidate optimizer scouts from the current model weights for $s$ epochs. The best scout continues for the remaining $b-s$ epochs, and that completed segment becomes the new incumbent if it improves the validation objective. This design allows the preferred optimizer to change as training progresses. We compare two variants of ROR on MNIST, Fashion-MNIST, and two motor insurance claim-count models. Nine fixed optimizers and both ROR variants are evaluated with the same ten seeds. One-epoch ROR uses 24\% to 35\% of the aggregate training needed to identify the best fixed optimizer exhaustively and remains close to that optimizer on all four tasks. These results support short scouting as a practical way to search over optimizers without completing every candidate run.
