针对适者生存训练中种群剪枝的未来性能临近预测
Nowcasting Future Performance for Population Pruning in Survival-of-the-Fittest Training
AI总结:
本研究提出用临近预测器预测模型未来准确率,结合适者生存(SOTF)训练进行种群剪枝,在ResNet-18、ViT等性能差异大的模型库中,可缩小与最优模型的性能差距。
摘要:
训练高性能神经网络通常需要评估具有不同超参数、架构或随机种子的候选模型种群。将每个候选模型训练至完成阶段计算成本高昂,但过早停止看似较弱的运行可能会丢弃最终表现更好的模型,因为学习曲线会发生交叉。为解决这一挑战,我们研究中间训练检查点是否可以预测模型的未来准确率,从而做出更明智的剪枝决策。我们的临近预测器可预测未来5个训练周期后的准确率或训练结束时的准确率,其以准确率与损失历史、权重统计量以及SANE表示作为输入进行训练。SANE是一种将检查点编码为潜在标记序列的Transformer自编码器。随后我们在适者生存(Survival-of-the-Fittest, SOTF)训练中使用这些临近预测器。SOTF在训练过程中根据预测准确率对运行进行排序,并停止排名最低的运行。我们的结果表明,权重统计量对5个周期的预测效果最佳,而SANE表示对最终准确率预测效果最佳。为进行评估,我们将SOTF与按当前准确率剪枝种群的方法进行比较。在同质模型库(如CNN模型库)中,按当前准确率剪枝已能保留最终模型,其性能仅略逊于最优模型。而在性能差异更大的模型库(如ResNet-18和ViT模型库)中,仅靠当前准确率进行剪枝更为困难。在这些模型库中,使用临近预测最终准确率的SOTF几乎能缩小与最优模型的性能差距。
英文摘要:
Training high-performing neural networks often involves evaluating populations of candidate models with varying hyperparameters, architectures, or random seeds. Training every candidate to completion is computationally expensive, yet stopping seemingly weaker runs early risks discarding models that would ultimately perform better, as learning curves can cross. To address this challenge, we investigate whether intermediate training checkpoints can predict future model accuracy and enable more informed pruning decisions. Our nowcasters predict test accuracy five epochs ahead or at the end of training. They are trained with accuracy and loss histories, weight statistics, and SANE representations as inputs. SANE is a transformer autoencoder that encodes a checkpoint as a sequence of latent tokens. We then use the nowcasters in Survival-of-the-Fittest (SOTF) training. SOTF repeatedly ranks runs during training by predicted accuracy and stops the lowest-ranked ones. Our results show that weight statistics work best for five-epoch forecasts, and SANE representations work best for final accuracy prediction. For evaluation, we compare SOTF with pruning the population by current accuracy. In homogeneous model zoos such as the CNN zoos, pruning by current accuracy already retains a final model only slightly worse than the best. Model zoos with a larger spread in performance, such as the ResNet-18 and ViT zoos, are more difficult to prune by current accuracy alone. In these zoos, SOTF with nowcasted final accuracy almost closes the gap to the best model.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。