EsaCL：稀疏模型的高效持续学习
EsaCL: Efficient Continual Learning of Sparse Models
AI总结：
本文提出EsaCL方法，通过方向性剪枝和智能数据选择策略，在无需重新训练的情况下高效持续学习稀疏模型，在多个基准上以更少资源达到与最先进方法相当的性能。
摘要：
持续学习场景中的一个关键挑战是高效地学习一系列任务，同时不遗忘如何执行先前学过的任务。许多现有方法要么在先前任务上重新训练模型，要么扩展模型以适应新任务。然而，这些方法通常面临存储和计算需求增加的问题，而在稀疏模型情况下该问题更加严重，因为稀疏化后需要昂贵的重新训练。为应对这一挑战，我们提出了一种用于稀疏模型高效持续学习的新方法（EsaCL），该方法能够自动剪除冗余参数而不对模型预测能力产生不利影响，并避免重新训练的需要。我们对参数剪枝下的损失景观进行了理论分析，并设计了一种方向性剪枝（SDP）策略，该策略利用损失函数相对于模型参数的锐度信息。SDP确保模型预测精度损失最小，从而加速每个阶段稀疏模型的学习。为加速模型更新，我们引入了一种智能数据选择（IDS）策略，该策略能够识别用于估计损失景观的关键实例，从而显著提高数据效率。实验结果表明，EsaCL在三个持续学习基准上取得了与最先进方法具有竞争力的性能，同时大幅减少了内存和计算资源的使用。
英文摘要：
A key challenge in the continual learning setting is to efficiently learn a sequence of tasks without forgetting how to perform previously learned tasks. Many existing approaches to this problem work by either retraining the model on previous tasks or by expanding the model to accommodate new tasks. However, these approaches typically suffer from increased storage and computational requirements, a problem that is worsened in the case of sparse models due to need for expensive re-training after sparsification. To address this challenge, we propose a new method for efficient continual learning of sparse models (EsaCL) that can automatically prune redundant parameters without adversely impacting the model's predictive power, and circumvent the need of retraining. We conduct a theoretical analysis of loss landscapes with parameter pruning, and design a directional pruning (SDP) strategy that is informed by the sharpness of the loss function with respect to the model parameters. SDP ensures model with minimal loss of predictive accuracy, accelerating the learning of sparse models at each stage. To accelerate model update, we introduce an intelligent data selection (IDS) strategy that can identify critical instances for estimating loss landscape, yielding substantially improved data efficiency. The results of our experiments show that EsaCL achieves performance that is competitive with the state-of-the-art methods on three continual learning benchmarks, while using substantially reduced memory and computational resources.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
