持续逻辑回归的紧凑记忆
Compact Memory for Continual Logistic Regression
AI总结：
针对持续逻辑回归的灾难性遗忘问题，提出基于海森矩阵匹配和概率PCA的紧凑记忆构建方法，大幅提升准确率，为持续深度学习的紧凑记忆研究开辟新方向。
摘要：
尽管最近取得了进展，持续学习的性能仍未达到批量训练的水平。为避免灾难性遗忘，我们需要构建过去关键知识的紧凑记忆，但即使对于仅含一两层的浅层神经网络，也尚未出现明确的解决方案。本文提出一种为逻辑回归构建紧凑记忆的新方法，该方法基于Khan和Swaroop [2021]的研究结果——他们证明了此类模型存在最优记忆。我们将最优记忆的搜索问题表述为海森矩阵匹配，并提出概率主成分分析方法进行估计。与经验回放相比，该方法能大幅提升准确率：例如在Split-ImageNet上，当记忆大小相当于数据规模的0.3%时，准确率达60%，而经验回放仅30%；将记忆大小增至2%时，准确率进一步提升至74%，接近批量训练的77.6%。该工作为构建紧凑记忆开辟了新方向，未来对持续深度学习也有帮助。
英文摘要：
Despite recent progress, continual learning still does not match the performance of batch training. To avoid catastrophic forgetting, we need to build compact memory of essential past knowledge, but no clear solution has yet emerged, even for shallow neural networks with just one or two layers. In this paper, we present a new method to build compact memory for logistic regression. Our method is based on a result by Khan and Swaroop [2021] who show the existence of optimal memory for such models. We formulate the search for the optimal memory as Hessian-matching and propose a probabilistic PCA method to estimate them. Our approach can drastically improve accuracy compared to Experience Replay. For instance, on Split-ImageNet, we get 60% accuracy compared to 30% obtained by replay with memory-size equivalent to 0.3% of the data size. Increasing the memory size to 2% further boosts the accuracy to 74%, closing the gap to the batch accuracy of 77.6% on this task. Our work opens a new direction for building compact memory that can also be useful in the future for continual deep learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
