基于Transformers的高效记忆持续学习
Memory Efficient Continual Learning with Transformers
摘要：
在许多现实场景中，用于训练机器学习模型的数据随着时间的推移变得可用。不幸的是，这些模型很难在不忘记过去所学知识的情况下不断学习新概念。这种现象被称为灾难性遗忘，由于实际的限制，很难预防。例如，可以存储的数据量或可以使用的计算资源可能是有限的。此外，应用程序越来越依赖于大型预训练神经网络，例如预训练的Transformers，因为资源或数据可能不足以大量地供从业者从头开始训练模型。在本文中，我们设计了一种方法，增量训练一系列的任务，使用预先训练的Transformers和扩展它们与适配器的模型。与现有的方法不同，我们的方法能够扩展到大量的任务，而没有显着的开销，并允许跨任务共享信息。在图像和文本分类任务中，我们根据经验证明，我们的方法保持了良好的预测性能，而无需随着时间的推移重新训练模型或增加模型参数的数量。与基于适配器的最先进方法相比，所产生的模型在推理时间上也明显更快。
英文摘要：
In many real-world scenarios, data to train machine learning models becomes available over time. Unfortunately, these models struggle to continually learn new concepts without forgetting what has been learnt in the past. This phenomenon is known as catastrophic forgetting and it is difficult to prevent due to practical constraints. For instance, the amount of data that can be stored or the computational resources that can be used might be limited. Moreover, applications increasingly rely on large pre-trained neural networks, such as pre-trained Transformers, since the resources or data might not be available in sufficiently large quantities to practitioners to train the model from scratch. In this paper, we devise a method to incrementally train a model on a sequence of tasks using pre-trained Transformers and extending them with Adapters. Different than the existing approaches, our method is able to scale to a large number of tasks without significant overhead and allows sharing information across tasks. On both image and text classification tasks, we empirically demonstrate that our method maintains a good predictive performance without retraining the model or increasing the number of model parameters over time. The resulting model is also significantly faster at inference time compared to Adapter-based state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
