迈出一步并环顾四周：关于使用增量数据进行热启动训练
Step Out and Seek Around: On Warm-Start Training with Incremental Data
AI总结：
本文针对顺序到达新数据时的模型持续改进问题，提出CKCA算法，通过FeatReg保留旧知识并利用AdaKD缓解遗忘和迁移知识，在ImageNet上相比普通热启动提升最高8.39%的top1准确率。
摘要：
在自动驾驶等现实世界的深度学习应用中，数据往往随时间按顺序到达。当新的训练数据可用时，从头开始训练模型会削弱利用已学知识的优势，导致显著的训练成本。从先前训练的检查点进行热启动是保留知识并推进学习的最直观方式。然而，现有文献表明这种热启动会降低泛化能力。在本文中，我们提倡热启动，但要迈出先前收敛点，从而在不损害先前知识的情况下更好地适应新数据。我们提出知识巩固与获取（CKCA），这是一种持续模型改进算法，包含两个新颖组件。第一，一种新颖的特征正则化（FeatReg），用于保留并精炼现有检查点中的知识；第二，我们提出自适应知识蒸馏（AdaKD），这是一种用于遗忘缓解和知识迁移的新方法。我们使用训练数据的多个划分在ImageNet上测试了我们的方法。我们的方法相比普通热启动实现了高达8.39%的top1准确率提升，并以较大幅度持续优于现有技术。
英文摘要：
Data often arrives in sequence over time in real-world deep learning applications such as autonomous driving. When new training data is available, training the model from scratch undermines the benefit of leveraging the learned knowledge, leading to significant training costs. Warm-starting from a previously trained checkpoint is the most intuitive way to retain knowledge and advance learning. However, existing literature suggests that this warm-starting degrades generalization. In this paper, we advocate for warm-starting but stepping out of the previous converging point, thus allowing a better adaptation to new data without compromising previous knowledge. We propose Knowledge Consolidation and Acquisition (CKCA), a continuous model improvement algorithm with two novel components. First, a novel feature regularization (FeatReg) to retain and refine knowledge from existing checkpoints; Second, we propose adaptive knowledge distillation (AdaKD), a novel approach to forget mitigation and knowledge transfer. We tested our method on ImageNet using multiple splits of the training data. Our approach achieves up to $8.39\%$ higher top1 accuracy than the vanilla warm-starting and consistently outperforms the prior art with a large margin.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
