可证明的对比持续学习
Provable Contrastive Continual Learning
AI总结：
本文为对比持续学习建立理论性能保证，揭示模型性能受先前任务训练损失约束，并据此提出使用自适应蒸馏系数的CILA算法，在标准基准上取得新的最先进性能。
摘要：
持续学习要求模型在动态数据分布下学习增量任务。迄今为止，已有研究观察到，在持续学习训练中结合对比损失和蒸馏损失能够取得强劲的性能。然而，据我们所知，这一对比持续学习框架缺乏令人信服的理论解释。在本工作中，我们通过建立理论性能保证来填补这一空白，这些保证揭示了在对比持续学习框架中，模型性能如何受先前任务训练损失的约束。我们的理论解释进一步支持了预训练可以有益于持续学习的观点。受这些保证的理论分析启发，我们提出了一种新颖的对比持续学习算法，名为CILA，该算法对不同任务使用自适应蒸馏系数。这些蒸馏系数可通过先前任务的平均蒸馏损失与平均对比损失之间的比率轻松计算。我们的方法在标准基准上显示出显著改进，并取得了新的最先进性能。
英文摘要：
Continual learning requires learning incremental tasks with dynamic data distributions. So far, it has been observed that employing a combination of contrastive loss and distillation loss for training in continual learning yields strong performance. To the best of our knowledge, however, this contrastive continual learning framework lacks convincing theoretical explanations. In this work, we fill this gap by establishing theoretical performance guarantees, which reveal how the performance of the model is bounded by training losses of previous tasks in the contrastive continual learning framework. Our theoretical explanations further support the idea that pre-training can benefit continual learning. Inspired by our theoretical analysis of these guarantees, we propose a novel contrastive continual learning algorithm called CILA, which uses adaptive distillation coefficients for different tasks. These distillation coefficients are easily computed by the ratio between average distillation losses and average contrastive losses from previous tasks. Our method shows great improvement on standard benchmarks and achieves new state-of-the-art performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
