SketchOGD：内存高效的持续学习
SketchOGD: Memory-Efficient Continual Learning
AI总结：
SketchOGD通过矩阵压缩技术实现内存高效的持续学习，解决了传统OGD在长期持续学习中的内存瓶颈问题。
摘要：
当机器学习模型在一系列任务上持续训练时，它们往往容易忘记之前任务所学的内容——这种现象称为灾难性遗忘。针对灾难性遗忘的解决方案通常涉及存储关于过去任务的信息，这意味着内存使用是决定其实用性的重要因素。本文开发了一种内存高效的解决方案来解决灾难性遗忘问题，利用矩阵压缩的思想，在一个称为正交梯度下降（OGD）的简单持续学习算法的背景下。OGD寻找旨在保持对先前数据点性能的权重更新，使用这些数据点上的模型梯度。然而，由于存储先前模型梯度的内存成本随着算法运行时间的增长而增加，OGD不适合长时间跨度的持续学习。为了解决这个问题，我们提出了SketchOGD。SketchOGD采用在线压缩算法，将遇到的模型梯度压缩成一个固定大小的矩阵。与现有内存高效的OGD变种不同，SketchOGD在线运行而不需要提前知道任务总数，易于实现，并且更易于分析。我们提供了关于相关压缩在下游任务OGD中适用的新型度量标准下的近似误差的理论保证。实验上，我们发现SketchOGD在给定固定内存预算的情况下，通常优于当前最先进的OGD变种。
英文摘要：
When machine learning models are trained continually on a sequence of tasks, they are often liable to forget what they learned on previous tasks--a phenomenon known as catastrophic forgetting. Proposed solutions to catastrophic forgetting tend to involve storing information about past tasks, meaning that memory usage is a chief consideration in determining their practicality. This paper develops a memory-efficient solution to catastrophic forgetting using the idea of matrix sketching, in the context of a simple continual learning algorithm known as orthogonal gradient descent (OGD). OGD finds weight updates that aim to preserve performance on prior datapoints, using gradients of the model on those datapoints. However, since the memory cost of storing prior model gradients grows with the runtime of the algorithm, OGD is ill-suited to continual learning over long time horizons. To address this problem, we propose SketchOGD. SketchOGD employs an online sketching algorithm to compress model gradients as they are encountered into a matrix of a fixed, user-determined size. In contrast to existing memory-efficient variants of OGD, SketchOGD runs online without the need for advance knowledge of the total number of tasks, is simple to implement, and is more amenable to analysis. We provide theoretical guarantees on the approximation error of the relevant sketches under a novel metric suited to the downstream task of OGD. Experimentally, we find that SketchOGD tends to outperform current state-of-the-art variants of OGD given a fixed memory budget.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
