打破滞后:用于增强优化的三重指数移动平均
Break a Lag: Triple Exponential Moving Average for Enhanced Optimization
AI总结:
针对现有一阶EMA优化器追踪复杂梯度趋势滞后、优化效果欠佳的问题,提出基于三重指数移动平均的FAME优化器,在30种CV架构的多任务基准测试中准确率和鲁棒性更优且可扩展性强。
摘要:
深度学习模型的性能关键取决于复杂的优化策略。现有优化器虽已展现出可观效果,但大多依赖一阶指数移动平均(Exponential Moving Average, EMA)技术,这往往限制了其准确追踪复杂梯度趋势的能力,会导致趋势识别出现显著滞后,优化效果欠佳,在梯度行为高度动态的场景中尤为明显。为解决这一根本局限,我们提出了快速自适应矩估计(Fast Adaptive Moment Estimation, FAME)——一种利用三重指数移动平均能力的新型优化技术。通过引入先进的追踪机制,FAME提升了对数据动态的响应能力,缓解了趋势识别滞后问题,优化了学习效率。我们开展了全面评估,涵盖图像分类、目标检测、语义分割等不同计算机视觉任务,将FAME集成到从轻量CNN到Vision Transformer的30种不同架构中。通过与当前最优优化器的严格基准测试,FAME展现出更优的准确率和鲁棒性。值得注意的是,它具备高可扩展性,在不同模型复杂度、架构、任务和基准上均带来显著提升。
英文摘要:
The performance of deep learning models is critically dependent on sophisticated optimization strategies. While existing optimizers have shown promising results, many rely on first-order Exponential Moving Average (EMA) techniques, which often limit their ability to track complex gradient trends accurately. This fact can lead to a significant lag in trend identification and suboptimal optimization, particularly in highly dynamic gradient behavior. To address this fundamental limitation, we introduce Fast Adaptive Moment Estimation (FAME), a novel optimization technique that leverages the power of Triple Exponential Moving Average. By incorporating an advanced tracking mechanism, FAME enhances responsiveness to data dynamics, mitigates trend identification lag, and optimizes learning efficiency. Our comprehensive evaluation encompasses different computer vision tasks including image classification, object detection, and semantic segmentation, integrating FAME into 30 distinct architectures ranging from lightweight CNNs to Vision Transformers. Through rigorous benchmarking against state-of-the-art optimizers, FAME demonstrates superior accuracy and robustness. Notably, it offers high scalability, delivering substantial improvements across diverse model complexities, architectures, tasks, and benchmarks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。