LoRanPAC：用于弥合持续学习中理论与实践鸿沟的低秩随机特征与预训练模型
LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning
AI总结：
针对持续学习领域预训练模型类方法缺理论保障、原理性方法性能不佳的问题，提出LoRanPAC方法，通过高维映射加SVD截断解决病态问题，兼具理论保证与SOTA性能。
摘要：
持续学习（CL）的目标是训练出能够解决按顺序呈现的多个任务的模型。近期的持续学习方法通过利用对下游任务具有良好泛化能力的大型预训练模型，取得了优异的性能。然而，这类方法缺乏理论保障，容易出现意外故障。相反，基于原理的持续学习方法往往无法达到具有竞争力的性能。在本研究中，我们旨在通过设计一种兼具理论合理性与高性能的简单持续学习方法，弥合这一理论与实践之间的鸿沟。具体而言，我们将预训练特征映射到更高维空间，并构建了一个过参数化的最小范数最小二乘问题。我们发现映射后的特征存在严重的病态问题，可能会导致较大的训练误差（数值不稳定）以及泛化误差上升。我们通过持续截断映射后特征的奇异值分解来应对这些挑战。我们提出的方法名为LoRanPAC，其性能对超参数的选择具有稳定性，能够处理数百个任务，并且在多个数据集上的表现优于当前最先进的持续学习方法。重要的是，我们的方法在整个持续学习过程中满足一种递推关系，这使得我们能够证明：通过适当地截断一部分SVD因子，该方法可以维持较小的训练误差与测试误差。由此得到的是一种兼具强劲实证性能与理论保障的稳定持续学习方法。代码可在以下地址获取：https://github.com/liangzu/loranpac。
英文摘要：
The goal of continual learning (CL) is to train a model that can solve multiple tasks presented sequentially. Recent CL approaches have achieved strong performance by leveraging large pre-trained models that generalize well to downstream tasks. However, such methods lack theoretical guarantees, making them prone to unexpected failures. Conversely, principled CL approaches often fail to achieve competitive performance. In this work, we aim to bridge this gap between theory and practice by designing a simple CL method that is theoretically sound and highly performant. Specifically, we lift pre-trained features into a higher dimensional space and formulate an over-parametrized minimum-norm least-squares problem. We find that the lifted features are highly ill-conditioned, potentially leading to large training errors (numerical instability) and increased generalization errors. We address these challenges by continually truncating the singular value decomposition of the lifted features. Our approach, termed LoRanPAC, is stable with respect to the choice of hyperparameters, can handle hundreds of tasks, and outperforms state-of-the-art CL methods on multiple datasets. Importantly, our method satisfies a recurrence relation throughout its continual learning process, which allows us to prove it maintains small training and test errors by appropriately truncating a fraction of SVD factors. This results in a stable continual learning method with strong empirical performance and theoretical guarantees. Code available: https://github.com/liangzu/loranpac.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
